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THE INFLUENCE OF PERSONALITY TRAITS, PRE-COLLEGE 
CHARACTERISTICS, AND CO-CURRICULAR EXPERIENCES ON 

COLLEGE OUTCOMES 

Karen W. Bauer 1 

Associate Director, Institutional Research & Planning 
University of Delaware 



Abstract 

The relationship between students ’ pre-college characteristics, personality traits and co- 
curricular activities with academic achievement and critical thinking was examined in a 
sample of 252 engineering, science, math, and psychology undergraduates enrolled at a 
selective Carnegie 1-Extensive doctoral-granting university. Results show that 
personality traits influence college outcomes both directly and indirectly through co- 
curricular activities even after controlling for pre-college characteristics, such as SAT 
score and high school GPA. Compared to personality traits, pre-college characteristics 
show larger effects on GPA and critical thinking skills. 

Acquisition of content knowledge and critical thinking are critical components of 
intellectual development as well as measures of college success. Among college 
outcomes, achievement has been one of the most frequently researched topics in higher 
education (Astin, 1977), and critical thinking skills are regarded as one of the major 
outcomes of college education (Pascarella, 1989; Facione, Sanchez, Facione, & Gainen, 
1995). College outcomes have been investigated using various predictors, including 
general verbal abilities, aptitude test scores (e.g., SAT and ACT), sex, family financial 
characteristics, in and out-of-class experiences, and personality traits (Astin, 1977, 1993; 
Ting & Robinson, 1998; Pascarella, 1989; Pascarella, Whitt, Edison, Nora, Hagendom, 
Yeager & Terenzini, 1996, 1997; Child, 1969; Entwistle & Entwistle, 1970, Digman & 
Takemoto-Chock, 1981). Although it is likely that personality traits are related to 
students’ participation in activities, and that activities also influence the outcomes, few 
studies have investigated these variables simultaneously. 

Among studies that investigated college outcomes using pre-college characteristics, 
SAT (or ACT) scores and high school GPA consistently explain the largest variance in 
college outcomes. In predicting the first year grade point average (GPA), high school 
GPA predicted the largest, unique variance (Ting & Robinson, 1998); in predicting 
critical thinking skills, general verbal ability explained the largest variance. Using the 
California Critical Thinking Skills Test (CCTST), Jacobs (1995) found that critical 
thinking is highly correlated with SAT verbal scores, and Ackerman (1999) reported a 
strong relationship between knowledge and general intelligence (g). Pascarella (1994) 



1 The author wishes to thank former postdoctoral researcher, Hye-Sook Park, for her assistance in data 
analysis for this project. 
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found that pre-college characteristics influence not only the outcomes of college directly, 
but also indirectly the outcomes through college course-taking activities, formal 
classroom experience, and out-of-class experiences. However, according to Mouw and 
Khanna (1990), the unique effect of pre-college predictors (e.g., high school GPA, 
college entrance tests) for college success was low. Due to a high correlation between 
these predictors, the unique contribution of each predictor on college outcome was small; 
thus, a large amount of variation on college GPA still needs to be explained. 

To investigate the unclaimed variance in GPA, some researchers have examined 
personality characteristics as an additional predictor of college performance (Tross, 
Harper, Osher, & Kneidinger, 2000). Findings on the predictive ability of personality 
characteristics are somewhat mixed; some researchers (e.g., Biggs, Roth, & Strong, 1970; 
Evans, 1970; Morgan, 1972) report that personality characteristics were not related to 
student GPA when aptitude characteristics such as SAT were controlled. Other 
researchers, however, found that personality type influences one’s activities (Hooker, 
Frazier, & Manahan, 1994) and, in particular, college behaviors and outcomes (Tross, et 
al, 2000; Digman & Takemoto-Chock, 1981; Entwistle, 1972). 

The effect of personality traits on achievement also varies depending on ability and 
age level. Entwistle’s (1972) review of studies involving Cattell’s 16 Personality Factors 
and Eysenck’s Personality Inventory concluded that success at the university level is 
associated with introversion, but at the primary school level, success is related to stable 
(low neuroticism) extroversion. According to Child (1969), both introversion and 
neuroticism are advantageous traits for university students’ academic achievement 
because introverts avoid social situations and enjoy bookish and abstract/conceptual 
pursuits, while neurotics have a higher level of internal drive. Entwistle & Entwistle 
(1970) partly attribute introverts’ higher academic achievement to their good study 
habits. Additionally, Digman & Takemoto-Chock (1981) found that conscientious 
students are well-organized, purposeful, and persistent, and that these characteristics are 
highly related to academic achievement (e.g., GPA). However, these studies did not 
control for students’ pre-existing aptitudes in investigating effect of personality on 
cognitive outcomes. Thus, it is difficult to justify the effect of personality traits on 
academic attainment objectively. 

Researchers have also studied how out-of-classroom experiences influence college 
students’ academic, intellectual or cognitive outcomes. For example, Inman & Pascarella 
( 1 998) found that college attendance positively affects the development of critical 
thinking skills. Other researchers found a positive association between the nature and 
frequency of students’ out-of-class contacts with faculty members and gains on measures 
of academic or cognitive development. For example, students’ participation in 
internships or study-abroad experiences was related to higher grades and to self-reported 
gains in knowledge of a particular discipline (Astin, 1993; Kuh, 1995). Also students’ 
out-of-class interactions contribute to gains in general knowledge, critical thinking skill 
and problem solving skills (e.g., Astin, 1993; Baxter-Magolda, 1992; Kuh, 1995; 
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Terenzini, Springer, Pascarella & Nora, 1995). Pascarella, et al. (1989) found that 
college experiences were modestly associated with higher critical thinking skills, while 
the composite college experience scale (e.g., type of course work, non-classroom 
interaction with faculty, study time, and extra-curricular activities) showed significant 
correlation with overall critical thinking skills. 

Despite some possible causal relationship among these variables, few studies have 
investigated the dynamic relationships among students’ personality types, college 
students’ co-curricular activities, and students’ cognitive outcomes. Thus, this study 
investigated how personality traits affect students’ involvement in co-curricular activities 
and college outcomes, both directly and indirectly via co-curricular activities. Research 
questions for this study are: 

1 . What is the relationship between first year students’ personality types, co- 
curricular activities, and end-of-first year outcomes (defined as end of first 
year GPA and critical thinking score)? 

2. Does personality type predict first year GPA and critical thinking score? 

3. Do pre-college characteristics predict first year GPA and critical thinking 
score? 

4. Do first year college co-curricular activities affect first year GPA and critical 
thinking score? 



Method 



Participants 

Participants were 264 undergraduate students who are part of a four-year longitudinal 
study funded by the National Science Foundation to assess academic and psychosocial 
effects of involvement in various college activities. The majority of students majored in 
science, engineering, and psychology at a Carnegie I Doctoral/Research-Extensive state 
university. Of the original sample, 252 students were included in this study. 2 Among 
them were 149 females (59%) and 103 males (41%), 193 White (76.5 %) and 59 non- 
Whites (23.5%). Table 1 shows the descriptive statistics of this sample. 

Table 1. 

Summary statistics and internal consistency estimates of CSEO Quality of Effort (OE) Subscales. 
NEO-Five Factor Inventory (FFI), and Watson-Glaser Critical Thinking Appraisal (VVGCTA). 
Variable N Mean SD Alpha 



CSEQ QE-Library Experiences Scale 


250 


19.61 


4.45 


0.75 


CSEQ QE-Experiences with Faculty Scale 


252 


19.68 


4.74 


0.83 


CSEQ QE-Course Learning Scale 


251 


27.79 


5.09 


0.80 


CSEQ QE-Art, Music, Theater Scale 


247 


18.48 


5.02 


0.75 


CSEQ QE-Student Union Scale 


252 


22.25 


5.36 


0.79 



2 



Among these 264 students, twelve students were not included because their factor scores on student 
union activity and arts were considered as outliers (3 SD above the mean). 

3 
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Variable 



CSEQ QE-Athletic/Recreation Facilities Scale 

CSEQ QE-Clubs and Organizations Scale 

CSEQ QE-Experience in Writing Scales 

CSEQ QE-Personal Experiences Scale 

CSEQ QE-Student Acquaintances Scale 

CSEQ QE-Science/Technology Scale 

CSEQ QE-Dormitory/Fraternity/Sorority Scale 

CSEQ QE-Topics of Conversation Scale 

CSEQ QE-Information in Conversation 

NEO-Neuroticism 

NEO-Extroversion 

NEO-Openness to Experience 

NEO- Agreeableness 

NEO-Conscientiousness 

Watson-Glaser Critical Thinking Appraisal 

Instruments 



N 


Mean 


SD 


Alpha 


250 


18.59 


5.87 


0.81 


250 


17.20 


6.37 


0.89 


251 


25.25 


6.14 


0.88 


252 


22.34 


5.28 


0.77 


252 


24.81 


5.81 


0.84 


246 


24.23 


6.64 


0.88 


240 


24.03 


5.56 


0.85 


246 


20.76 


5.14 


0.80 


248 


13.97 


3.06 


0.76 


251 


22.10 


8.08 


0.86 


251 


30.36 


6.47 


0.81 


251 


30.12 


6.07 


0.73 


251 


32.40 


5.98 


0.77 


251 


32.65 


6.69 


0.83 


252 


29.54 


5.33 


0.77 



Information about these students were collected from the university’s student records 
database and by using the following three published measures: the NEO-Five Factor 
Inventory (NEO-FFI; Costa, Jr.& McCrae, 1991), the Watson-Glaser Critical Thinking 
Appraisal (WGCTA; Watson & Glaser, 1994), and the College Student Experiences 
Questionnaire (CSEQ; Pace, 1984). 

NEO-Five Factor Inventory (NEO-FFD. The NEO-FFI measures the most basic 
dimensions underlying human traits. There are five subtests in this inventory, each 
composed of twelve items. The five subtests are: 1) NEO-Neuroticism measures an 
individual’s level of adjustment and emotional stability (coefficient alpha= 0.86, n=12); 
2) NEO-Extroversion measures level of sociability and consequent behaviors that occur 
as a result of interactions with others (coefficient alpha= 0.81, n=12); 3) NEO-Openness- 
to-Experiences measures imagination, aesthetic sensitivity, attentiveness to inner feelings, 
preference for variety, intellectual curiosity, and independence of judgment (coefficient 
alpha= 0.73, n=12); 4) NEO-Agreeableness measures level of sympathy and 
altruism toward others, eagerness to help (coefficient alpha= 0.77, n=12); and 5) NEO- 
Conscientiousness measures ability to manage impulses and desires and the process of 
planning, organizing, and carrying out tasks (coefficient alpha= 0.83, n=12; Costa & 
McCrae, 1991). 

Watson Glaser Critical Thinking Appraisal (WGCTA). The WGCTA is a composite 
measure that examines attitudes of inquiry, knowledge of the nature of inferences, 
abstractions, and generalizations; and skills in employing the above attitudes and 
knowledge (Watson & Glaser, 1994). The WGCTA Form S consists of 40 items 
measuring five subtests of critical thinking: inference ; recognition of assumptions'. 
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deduction', interpretation; and evaluation of arguments. The WGCTA data for this 
sample has a reliability of 0.77. 

College Student Experiences Questionnaire (CSEQ). The CSEQ examines students’ 
quality of effort put forth with various college activities, level of satisfaction with the 
campus environment, perceptions of the campus environment (emphasis on scholarly, 
aesthetic, and vocational issues), and perceived annual gain in a series of academic and 
personal items. The CSEQ is composed of 14 quality of effort composite scales that 
measure level of student engagement, seven questions that query perceptions of the 
academic environment, and 21 items related to academic and social growth during the 
current college year. The reliability of the scores on the quality of effort scales ranged 
from 0.75 to 0.91 with an average of 0.85 

Procedure 

After receiving approval from the University’s Human Subjects Committee, 
researchers sent a letter to freshmen level students majoring in science, math, and 
psychology requesting their participation in a study of their academic experiences. Two 
hundred sixty-four students agreed to participate and met with the researchers to 
complete several questionnaires including the CSEQ, WGCTA, and NEO-FFI. Each 
survey took approximately 15 to 30 minutes to complete. A maximum of 30 minutes was 
allowed to complete the WGCTA. A signed consent form also enabled the researchers to 
obtain demographic data from the university’s student record system (i.e., high school 
GPA, sex, ethnic classification, SAT, and cumulative GPA). Students were given $5.00 
for their participation in this study. 

Data Preparation 

Four students’ SAT scores were not available (matriculated from a foreign country), 
and two students had extremely low scores and thus not included. Imputed values were 
created for the missing cases in order to maintain the sample size. In the CSEQ Quality 
of Effort subscales, some patterns of missing values were found. It seemed that students 
who thought the items were not directly related to them simply did not respond to these 
items. In order not to treat these as missing values at random or deleting these cases, 
imputed values were created by assigning the lowest value found in each item, 3 assuming 
that non-responding students simply did not respond or skipped the items. 

Three CSEQ composite scores based on conceptually-related items were created and 
treated as endogenous variables 4 : 1) academic activities, items related to academic and 
cognitively oriented activities: experience in writing, library use, course learning, and 



3 In the Amos program, bootstrapping was not possible with missing values. We imputed these missing 
values not to reduce the sample size (n=16). 

4 Principal component exploratory factor analysis of the CSEQ quality of effort subscales was attempted, 
but the screen plot showed only one factor. 
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experience with faculty; 2) conversation, items: topics of conversation, information in 
conversation, and personal acquaintance; and 3) club/union, items: club, student union, 
and campus residence activities. 



Results 

A non-recursive path analysis model was built to examine how personality traits 
influence college outcomes both directly and indirectly through mediation of co- 
curricular activities. See Figure 1 below. 




Figure 1. The Influence of Personality Traits and College Activities on College Outcomes 



In this model, personality trait scores, SAT score, high school GPA, interaction 
between SAT scores and high school GPA, and sex were used as exogenous variables; 
co-curricular activities served as mediating endogenous variables; Watson-Glaser critical 
thinking skills score and spring ’97 GPA were used as endogenous variables. The 
interaction effect was investigated by grand-mean centering of each predictor to avoid the 
problem of mulitcollinearity. An interaction effect between SAT total score and high 
school GPA on spring ’97 GPA was significant, so interaction terms were incorporated 
into the path model. Table 2 shows the correlation among variables in the model. 

The path model indicated a good fit (RMSEA= 0.045 5 ; NFI = 0.997; TLI 6 =0.995; %*= 
34.584; df = 23; p = 0.057; n=252). 7 The Amos 4.0 (Arbuckle, 1998), which employs 



5 90 percent Cl of RMSEA ranged from 0.00 to 0.074. 

6 The Tucker-Lewis Index (TLI) is also known as the Bentler-Bonett non-normed fit index (NNFI). 

6 
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maximum-likelihood estimates of parameters, is generally robust in the violation of 
assumptions with a simple model. However, due to the complexity of our model, and to 
avoid the violation of normality assumption, we deleted those outliers that were more 
than three standard deviations above the mean in each variable. The model without 
outliers indicates a better fit compared to the one with outliers, therefore these parameters 
can be interpreted with confidence. 



Table 2 

Correlation Matrix of the Variables in the Path Model 





Hgpa 


SAT 


Sex 


N 


E 


0 


A 


c 


Conver 


Acad 


Union 


WG S97gpa 


Hgpa 


— 
























SAT 


0.38** 


— 






















Sex 


-0.09 


0.26** 


— 




















N 


0.02 


-0.09 


-0.11 


— 


















E 


-0.01 


-0.17** 


-0.26** 


-0.35** 


— 
















0 


-0.08 


0.06 


0.00 


-0.02 


-0.02 


— 














A 


0.10 


-0.08 


-0.28** 


-0.13* 


0.28** 


0.06 


— 












C 


0.19** 


-0.16* 


-0.18** 


-0.24** 


0.21** 


-0.03 


0.13* 


— 










Conver 


-0.13* 


-0.19** 


-0.21** 


0.05 


0.22** 


0.41** 


0.08 


- 0.11 


— 








Acad 


0.09 


-0.17** 


-0.18** 


-0.17** 


0.19** 


0.44** 


0.10 


0.42** 


0.49** 


— 






Union 


0.09 


0.00 


-0.20** 


-0.12 


0.29** 


0.06 


0.09 


0.20** 


0.43** 


0.39** 


— 




WG 


0.16** 


0.63** 


0.24** 


-0.06 


-0.18** 


0.17** 


0.04 


-0.23** 


-0.16* 


-0.15* 


0.19** 


— 


S97gpa 


0.48** 


0.39** 


0.07 


0.00 


-0.17** 


0.00 


0.05 


0.21** 


-0.15** 


0.15* 


-0.06 


0.25** 



Note: **:p<0.01, *: p<0.05 

Hgpa represents high school GPA; Conver reperesents CSEQ conversation-related activities; Acad 
represents CSEQ academic activities; and Union represents CSEQ student union/club/campus residence- 
related activities. 

Personality Traits Influence Critical Thinking Skills and Spring ’97 GPA 

The effect of students’ personality traits on students’ cognitive outcomes were 
statistically significant even after controlling for the pre-college characteristics. As 
shown in Tables 3 and Figure 1, the effect of NEO-Openness-to-Experiences on WGCTA 
was significant. This means that when the effects of other predictors in the model were 
controlled, a one standard deviation increase in NEO-Openness-to-Experiences scale was 
related to a 0. 138 standard deviation increase in WGCTA. In addition, NEO- 
Agreeableness had a positive effect on WGCTA, and NEO-Conscientiousness also had a 
positive effect on spring ’97 GPA. However, NEO-Ex trovers ion had a negative effect on 
spring ’97 GPA. 



7 A just identifiable (saturated) model with 1 19 parameters had the value of 1 in NFI and p-value, and 
0.948 in ECVI, while the ECVI of our default model with 96 parameters was 0.903. 
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Table 3. 

Standardized Effect of Personality on Grades and Critical Thinkin2 Skills 



Outcome 


Neuroticism 


Extroversion 


Openness 


Agreeableness 


Conscientiousness 


Spr’97GPA 

Direct 

Indirect 


-0.020 


-0.111* 

-0.045 


0.046 


0.000 


0.152** 

0.050 


Total 


-0.020 


-0.156 


0.046 


0.000 


0.202 


WGCTA 

Direct 

Indirect 


0.000 


-0.043 


0.138** 

-0.008 


0.124* 

0.000 


-0.020 


Total 


0.000 


-0.043 


0.129 


0.124 


-0.020 



** : jk. 01, *: £<.05; represents parameters are not obtained 



Influence of Pre-college Characteristics 



In addition to personality traits, pre-college characteristics were found to have a 
significant effect on academic and cognitive outcomes. The effect of SAT score on 
WGCTA was 0.632 (p<0.001), which means that when holding all other variables in the 
model constant, a one standard deviation increase in total SAT scores was related to a 
0.63 standard deviation increase in WGCTA. (The standardized path coefficient of sex 
on WGCTA was not significant.) The effect of total SAT score on spring ’97 GPA was 
0.295 (p<0.01) and the effect of high school GPA on spring ’97 GPA was 0.343 (gcO.Ol). 
There was also an interaction effect of high school GPA and SAT on the spring ’97 GPA, 
which indicates that the effect of SAT scores on the spring ’97 GPA depends on students’ 
high school GPA. 



Additionally, the model with only three pre-college characteristics (i.e., SAT scores, 
high school GPA, and sex) explained 40 percent of the variance in WGCTA, and eight 
percent of variance 8 in the spring ’97 GPA. A simple regression analysis using 
personality characteristics and activities yielded a model that explained 15 percent of the 
variance of the spring ’97 GPA and 17.5 percent of the variance of the WGCTA 9 . Thus, 
incorporating personality characteristics and co-curricular activities into the model was 
appropriate. 



8 When a simple regression was run using three pre-college characteristics (i.e., sex, high school GPA, and 
SAT), they explained about 29 percent of variance in spring ’97 GPA. 

Using only personality traits as independent variable, the model explained 10 percent of variance in 
spring 97 GPA and 13.5 percent of variance in WGCTA respectively. The three co-curricular related 
activity variables explained 9 percent of variance in spring ’97 GPA and 5 percent of variance in 
WGCTA respectively. 
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Personality Traits Influence Students’ Engagement in Co-Curricular Activities 

In addition to influencing critical thinking and grades. Table 4 shows that personality 
traits influence students’ engagement in co-curricular activities. Students with high 
scores on the NEO-Neuroticism scale were less likely to be engaged in academically- 
oriented activities. Students who scored high on the NEO-Extroversion scale were more 
likely to be engaged in club and student union-related activities and to engage in 
social/interpersonal communication (conversation)-related activities. Students who 
scored high on the NEO-Openness-to-Experience scale were more likely to spend time in 
academic activities, but they were also more likely to spend time engaging in 
social/interpersonal communication (conversation)-related activities. Students who 
scored high on the NEO-Conscientiousness scale were more likely to engage in 
academic/leaming-related activities and also were more likely to engage in student 
union/club activities. 



Table 4. 

Standardized Effects of Personality on Co-Curricular Activities 





Neuroticism 


Extroversion 


Openness Agreeableness 


Conscientiousness 


Academic 


-.099* 




.360** 


.328** 


Convers 

Union 




.164** 

224** 


394 ** 


.115** 


** : p< . 01 , 


*: p< .05; 


represents parameters are not obtained 





Results of the path model also showed a sex difference in conversation-related 
activities (P=-0.144, p<0.01). Females were more likely to participate in 
conversation/personal acquaintance-related activities. In addition, students with higher 
GPAs were more likely to engage in academically-oriented activities (P=0.159, p<0.01), 
but students with high SAT scores were less likely to engage in academically-oriented 
activities (P=-0.185, p<0.01). 

Direct Effect of Co-curricular Activities on Cognitive Outcomes 

As shown in Table 5, there was also a significant relationship between involvement in 
college activities and cognitive outcomes. Students’ engagement in club/union/campus 
residence activities was associated with lower scores in both WGCTA and spring ’97 
GPA. The effect of conversation-related activities on WGCTA and spring ’97 GPA did 
not show any statistical significance. However, academically-oriented activities showed 
a positive effect on spring ’97 GPA. These results indicate that participation in student 
union/club/activities has a negative effect on both WGCTA score and GPA. 
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Table 5. 



Standardized Effect of Activities on Cognitive Outcomes 



Outcome 


Academic 


Conversation 


Club/Union 


Spring 97GPA 


.205** 


-.072 


-. 149** 


WGCTA 




-.021 


-.176** 


* * : p< .01 *: p<.05; 


represents parameters are not obtained 

Discussion 





This study explores the relationship among students’ pre-college characteristics, 
personality traits, co-curricular activities, academic achievement, and critical thinking 
score. Results indicate that personality does influence students’ achievement and critical 
thinking. NEO- Agreeableness and NEO-Openness-to-Experience were positively and 
significantly related to WGCTA. Results suggest that students who are more extroverted 
will earn lower grades than peers who are less extroverted, and those who are more 
conscientious will earn higher grades than peers who are less conscientious. Unlike 
previous studies involving college students (Child, 1969), neuroticism does not seem to 
be a driving force for attaining high GPA. Additionally, since extroverted students were 
more likely to spend time on clubs and student-union related activities, it is possible that 
students who obtained high scores on the extroversion scale may devote less time to 
study which might lead to lower GPAs. 

After controlling for academic aptitude and personality traits, only academically- 
related co-curricular activities were positively and significantly related to GPA. This 
finding underscores the importance of students’ involvement in academic activities 
because academically-oriented activities contribute to higher GPAs. Note also that the 
effect of club/student union/campus residence-related activities on WGCTA was 
negative, a finding similar to that of previous studies (Pascarella et al, 1996). In addition, 
different types of activities influence the two academic outcomes differently, which may 
indicate the two college outcomes are measuring different things. 

Overall, the effects of pre-college characteristics such as SAT scores and high school 
GPA were larger than any other predictors for college outcomes. This result confirms 
Ting and Robinson’s findings (1998). The effect of co-curricular activities such as 
academic and club/union/residence hall-related activities were larger than the direct effect 
of personality traits on WGCTA and GPA with the exception of NEO-Conscientiousness 
on spring 97 GPA. This finding indicates that, for this sample, the effects of personality 
traits were relatively minor. However, personality traits influence college outcomes via 
students’ engagement in co-curricular activities because the effects of personality on 
students’ engagement in co-curricular activities were moderately high. 
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Implications for Faculty and Administrators 

Results from this study broaden previous findings on the relationship between 
personality type and college outcomes (Entwistle & Entwistle, 1970; Entwistle, 1972; 
Digman & Takemoto-Chock, 1981). In addition to pre-college characteristics reflected in 
SAT, high school GPA, and co-curricular activities, this study indicates that personality 
traits have a relatively small (but significant) effect on college outcomes of GPA and 
measure of critical thinking. However, personality has relatively larger impact on 
students’ engagement in co-curricular activities, which in turn influence academic 
outcomes directly. Thus, knowledge of personality traits may enable faculty and staff to 
facilitate students’ learning in an effective way. For example, students who score high in 
NEO-Extroversion or NEO-Openness-to-Experience are likely to explore such programs 
as undergraduate research, study-abroad, or major-related internships. More so than 
other students, student leaders, for example, may achieve cognitive and affective benefit 
due, in part, to their level of extroversion or conscientiousness. Student knowledge of 
their own personality traits can help make wise choices about college activities. 

Freshman year curricular and co-curricular choices can act as a scaffold to further 
students’ breadth of experiences and consequent increases in critical thinking skills. 

With knowledge of students’ personality scores in hand, faculty and advisors can 
suggest activities that achieve a good fit, or conversely, recommend activities that may 
not match student’ personality. For example, a student who scores highly on extroversion 
may thrive in public speaking activities whereas another who scores low on this trait will 
not. Similarly, a student who scores low on Openness-to-Experience will not likely enjoy 
nor benefit from study abroad or organizing a new student club. 

Accessibility to these activities would likely influence students’ engagement in co- 
curricular activities and further cognitive outcomes. Since measurable cognitive gains 
increase gradually over a number of years (Ackerman, 1999), it is also important that 
college officials help students understand that a variety of activities nurture cognitive 
growth and thus encourage students to become or remain active in volunteer community 
service, research with faculty mentors, and/or participate in major-related internships 
throughout their baccalaureate experience. 

Limitations 

Limitations of this study are related to external validity and length of study. Because 
of the self-selected nature of participants, the sample was not randomly selected, thus, 
limiting generalizability. 10 Due to the self-report nature of data, responses on the survey 
may not accurately convey their efforts in all activities. Since some of the activities are 



10 Based on one-sample t-test using SAT scores, all students except those in animal science, civil 
engineering, and psychology department were representative samples of the department. 
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socially more desirable than others, it is possible that students might choose those 
activities based on social acceptability rather than true interest. 

Finally, this study examines the relationship between personality traits, students’ 
engagement in activities, and college outcomes during first year of baccalaureate studies. 
Thus, it is not known if students’ engagement in college activities is a continuation of 
their high school activities, nor if the same co-curricular activities affect college 
outcomes throughout the baccalaureate experience. Additionally, it is also not known 
how and whether personality traits change over time and affect students’ engagement in 
activities differently. Thus, it would be more meaningful if similar research questions are 
investigated in a longitudinal fashion that employs growth modeling. 
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Introduction 

The degree-conferring rates of four-year colleges and universities typically and 
substantially outstrip those of two-year postsecondary institutions — at the national level, 
for example, by 65 to 23 percent respectively (Adelman, 2000). In this study, we posit a 
key role for remedial education in the formation of this “graduation gap”. Specifically, we 
reason that high developmental program participation rates plus low program completion 
rates tend to produce inflated attrition rates, especially at schools where remediation 
program completion is a prerequisite for enrolling in most entry-level credit courses. That 
attrition may be the overt sort (early college exiting), but here we mostly had in mind 
sizable numbers of what may be called “stealth dropouts”, continuing students who are 
remedial non-completers and therefore effectively precluded from the degree track 
course-taking. 

The national data fit the pattern in a general way: over two-fifths (41 percent) of all 
first-time freshmen entering public two-year schools in 1995 were enrolled in courses 
designed to remediate college skills deficits (National Center for Educational Statistics, 
1996), only 43 percent of such developmental students completed all their program 
requirements, also, at mid-decade more than half of the country’s community colleges 
mandated in-coming student developmental education placement testing and had 
established enrollment procedures essentially limiting serious credit course-taking to 
those who had finish remediation or required none (McCabe, 2000). 

Hardly any research, however, has been specifically devoted to exploring the 
interplay between remedial education skills-credentializing function and academic 
outcomes from a process perspective. In fact, little research attention of any kind has 
been paid to working out the details of developmental education as a process. Instead, 
most developmental research has tended to concentrate on practical institutional case 
studies concerning the salutary impact of specific program reforms (see Ignash, 1997; 
Boylan, 2000), although one does run across the occasional report on correlations 
between student degree progress and developmental program participation conceived 
mostly as an undifferentiated phenomenon (for example, Brophy, 1984; Keller and 
Williams-Randall, 1998; Yang, 2000; Zhao, 1999). 
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This study, in a small way, seeks to advance the understanding of developmental 
education as a process — a goal-organized dynamic of academic policies and instructional 
operations — capable of exerting an influence on academic outcomes comparable to the 
impacts of factors such as scholastic ability, and academic and social environments. This 
we hoped to accomplish demonstrating how the complex nature of the developmental 
program at one fairly representative community college systematically interacted with 
remedial student decisions and behaviors to limit access to degree programs. 

Institutional Setting and Developmental Program Characteristics 

Prince George’s Community College is a public, two-year postsecondary education 
provider in the Maryland suburbs of the District of Columbia, with a fiscal year credit 
enrollment averaging around 15,000 students. Its institutional performance in terms of 
state standard assessment indicators falls within the normal range for its peer group, and 
it is also unexceptional for a school of its type in the socio-economic composition of its 
student body, except for a very high concentration of African American attenders (70 
percent). It is fairly representative, as well, of state community colleges in the size and 
program area distribution of its remedial student enrollment, and in the form and 
functioning of its developmental education process (Maryland Higher Education 
Commission, 1996). 

All incoming credit course students are expected to undergo the full battery of 
remediation placement tests (DTLS/MS for three basic skills areas — English 
composition, reading comprehension and high school-level mathematics), or to seek and 
obtain formal exemptions based on prior college work (transfer students), scores received 
on national education tests (SAT or ACT), or past fulfillment of special preparatory 
programs (for example, a pre-registration intensive algebra review course). Students 
evading one or all remedial assessment are not formally prohibited from attempting credit 
enrollment, but will find this quite difficult in practice, lacking proof of basic skills 
proficiency that is a prerequisite for taking most entry-level credit courses. 

Area program courses fall into low tier-high tier sequences (with an intermediate tier 
in math), based on the number and type of skills deficiencies identified during the 
placement testing. Remedial students are placed into the appropriate tier given their test 
scores, and if placed into a lower tier must work their way up, with retrograde motion 
also a possibility under some conditions. Students with single area requirements who 
place into the top tier have a total developmental “course burden” of 1 ; those requiring the 
most intensive level of remediation in all three areas start out with a minimum course 
burden of 7. Developmental courses may be repeated only once, a non-advancing the 
second time around constituting formal program failure (although a peremptory course 
withdrawal is possible up to semester midpoint). For the most deficient, the course 
burden may reach 14. 
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Only institutional CEUs are awarded for passed developmental courses, but there 
exists no bar to students taking credit courses simultaneously with remedial ones, 
provided they meet course prerequisites, including those relating to basic skill 
proficiency. This means that students with yet-unremediated deficiencies in one area but 
not another are perfectly free to take credit courses which lack skills prerequisites of the 
first kind. Although students are recommended to finish their remedial studies early, no 
remediation schedule is mandated and they are free to enroll in developmental courses at 
any time or in any area order they choose. In fact, as in the case of test avoidance, failure 
to begin area programs on a timely basis or at all, or for that matter to complete any 
begun, does not preclude credit enrollment, subject to the usual course prerequisite 
caveats. The degree track is never denied to incompletely remediated students by formal 
prohibition. What effectively bars them is the way the system of credit course 
prerequisites works at registration. All entry courses to the general education program 
that students must be completed before graduating, and most degree program entry-level 
courses, as already noted, require proof of proficiency. 

Methodological Considerations 

It was important to go into such detail concerning developmental program procedure 
at PGCC because that is where the Devil is and where we had to start from in designing a 
study which, after all, puts procedural complexity at the center of research. In preparation 
for our work, a massive developmental program file was assembled based 1995-2000 
student transcript data, covering all the aspects of remediation procedure at PGCC just 
reviewed, plus student development program placements, decisions on program options, 
course behaviors, program outcomes and overall academic outcomes. The 
methodological approach adopted was longitudinal analysis, in this case of the cohort of 
all 1996 fall-entering first-time credit students (N=2,094). Cohort Fall-96 was the first for 
which developmental data was 100 percent complete and verified, and was also the first 
to feel the effects of the College’s new computer-driven course lock-out system designed 
to eliminate course prerequisite violations at registration. Using this cohort would also 
allow a sufficient time span (four years) for development and overall academic outcomes 
to become manifest. 

The next step was the choice of a developmental status measurement method 
appropriate to our research aims. The candidates were the conventional method, used in 
most institutional reporting, which sorts students solely on the basis of actual 
developmental placement testing and course-taking results, versus a new approach we 
call the degree-track method, keyed to ability of students to meet the skills proficiency 
prerequisites of degree-relevant credit courses. Basically, the former is more data-audit 
defensible but, as Table 1 below shows, tends to distort the meaning and inflate the size 
of the “not required” and “required/completed” categories by including within them 
students who skipped one or two placement tests and therefore who lack credentials in 
some skill areas. 
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Table 1. Two Remedial Status Measurement Methods ( Cohort Row Percentages) 


Method 


NOT REQ 


REQ 


REQ/CMP 


REQ/INC 


NO DATA 


Conventional 


34. 6 a 


58. 5 b 


14.4 C 


44. l d 


6.8 e 


Degree-Track 


25. 0 f 


75. 0 g 


13.T 


61.9 h 




a. no REQ any testing b. Any tested REQ 


c. all REQ/CMP d. Any REQ/INC e. Untested 


f. No REQ/all 3 tests g. any REQ/all 3 tests, or test miss h. Any REQ/INC or any test miss 



The latter, however, defines developmental non-completion as the inability to meet 
credit course prerequisites for basic skills proficiency, either because a student has failed 
to pass a required remediation program, or because one or more of his skills remains 
unassessed (missed placement tests). Accordingly, only students tested in all three skills 
area can sort into non-developmental or completed remediation categories, which is 
equivalent to placing them in a more general degree track category. Since the focus of our 
study was the linkage between the developmental process and access to the degree track, 
the second remedial status measurement method was obviously superior for our purposes 
and was used in all subsequent analysis. 



Table_2. 1996. Cohort 4-Year Developmental and Academic Outcomes (Percents) 



Selected Outcome 


Cohort 


DT 

Non- 

Dev. 


DT 

CMP 


Conv 

CMP 


DT 

INC 


Conv 

INC 


Conv 

No 

Tests 


Transfer Only 


8.2 


17.0 


9.1 


14.3 


4.4 


4.9 


2.1 


Both Deg./Transfer 


.9 


3.4 


.4 


mm 


.0 


.0 


.0 


Degree Only 


4.6 


10.1 


15.7 


inn 


.0 


,0 


.0 


Pre-Grad (45-59 Hrs) 


9.0 


12.0 


16.1 


39.3 


5.6 


7.9 


.7 


Soph (30-44 Hours) 


9.3 


11.9 


17.1 


7.1 


13.7 


19.1 


4.2 


Frosh (1-29 Hours) 


48.5 


38.6 


37.6 


39.3 


53.5 


52.3 


62.2 


No Credits Earned 


19.5 


6.9 


4.0 


.0 


22.8 


15.8 


30.8 


Column N 
Row 


2,094 


523 


274 


28 


822 


304 


143 


100.0 


25.0 


13.1 


1.3 


39.3 


14.5 


6.8 



DT=by degree-track method (all tests taken) Conv=conventional (hidden test 
skipping) 



Finally, before proceeding to the analytic phase of our study, we thought it prudent to 
make a reality check on the effectiveness of the college’ enrollment prerequisite structure 
in blocking skills-unproven students from the degree track. Table 2 (above) presents the 
results of that trial — a cross-tabulation of a variable dividing developmental status 
categories into degree track and non-degree track (conventionally defined) sub-types, 
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with selected four-year academic outcomes, including, most crucially, degree attainment. 
If a tight linkage exists between unproven skills proficiency and minimization of credit 
enrollments, then the percentages for table cells (deep shaded) representing degree 
attainment levels among off-track developmental status students should be zero or near to 
it, which proves to be the case. Even test-skipping “completers”, the most likely to break 
the pattern, collectively failed to earn a single associate degree or occupational certificate 
after four years of trying, while 16 percent of the true completers (all skills assessed/all 
program requirements fulfilled) managed to graduate. Interestingly, however, off-track 
completers accomplished more transfers, the single standard form of academic success 
procedurally open to them, than any other group. They also tended, appropriately, to have 
the largest numbers backed up in the near-graduation slot (39 percent). The only 
“degreeless” degree-track category (light shaded award results) properly turned out to be 
the all-test incompletes. 



Developmental Program Area Findings 

Our analytic work began with an exploration of what happened to cohort 
developmental students in each of the three skills remediation programs. The full range of 
the exhibited decisions, behaviors and outcomes of the sub-cohort within each remedial 
area was examined for patterns, leading to development of three master “area career path” 
variables. Each variable category represented a discreet career path through (or around) 
the remedial area program. Appropriate to the complex possibilities latent in the 
dynamics of postsecondary remedial education, these variables expressed a very large 
number of realized career paths: 28 paths in the cases of developmental English (DVE) 
and reading (DVR), and 47 in the case of developmental math (DVM). Table 3 (above) 
presents three manageably condensed versions of these variables along with their all- 
cohort, area-required student and area course-taking student percentage distributions. 



Table 3. Comparative Developmental Program Main Effects (Percentages) 





DVE 


DVR 


DVM 




ALL 


REQ 


CRS 


ALL 


REQ 


CRS 


ALL 


REQ 


CRS 




2094 


968 


475 


2094 


824 


419 


2094 


1,371 


642 


Not Required 


53.8 






60.7 


/■' 




34.5 : 


Required 


46.2 


100.0 




39.4 


100.0 


\ '-.i.,.' • . ; 


65.5 


100.0 




Completed 


16.0 


34.5 


68.0 


12.5 


31.7 


59.2 


11.7 


17.9 


34.9 


■ Course Pass 


15.1 


32.7 


66.7 


11.5 


29.2 


57.5 


10.2 


15.5 


32.2 


■ Re-Test Out 


.6 


1.3 


.2 


.8 


2.1 


1.0 


1.2 


1.8 


.5 


■ Late Start 


.2 


.5 


1.1 


.1 


.4 


.7 


.4 


.6 


1.2 


Incompletes 


30.2 


65.5 


32.0 


26.9 


68.3 


40.8 


53.7 


82.1 


65.1 


■ Unassessed 


15.3 


33.1 




13.7 


34.8 




16.0 


24.2 




■ No Courses 


7.7 


16.6 




5.0 


12.7 




17.8 


27.2 




■ NP Grade 1* 


2.2 


4.7 


9.9 


2.4 


5.8 


11.9 


7.1 


9.9 


23.1 


■ NP Grade2* 


.2 


.6 


.8 


.2 


.9 


1.2 


1.8 


3.7 


5.9 


■ Dropout/W 


3.8 


8.2 


16.6 


2.9 


7.4 


14.6 


7.8 


11.9 


25.4 


■ Dropout/P 


.9 


1.9 


3.8 


2.5 


6.4 


12.6 


2.7 


4.26 


8.9 


■ Late Start 


.2 


.4 


.8 


.1 


.2 


.5 


.6 


.9 


1.9 



*NP Grade 1=N on -pas sing/ 1st Attempt; NP Grade2 = Non-passing/2nd Attempt 
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The table is rich in findings and policy implications, but given limited room for 
explication, we will focus on those which bear directly on the central concern of this 
paper — student degree track preclusion. This means looking mostly at the remediation 
incomplete categories. Inevitably, students who fell into any of these would be unable to 
meet the skill proficiency meet of at least one key credit course need for degree program 
advancement. Any incomplete career path cohort percentages ranged from a substantial 
27 percent (DVR) to a majority 54 percent (DVM). Such high area rates were 
unsurprising, considering the overall remediation incomplete figure we saw earlier (62 
percent). But what was unexpected was how relatively little program incompleteness was 
attributable to poor student developmental course performance. For example, only 14 
percent of the students needing skills credentials in math (DVM/REQ) failed to complete 
remediation due to non-passing DVM course grades in their last course (first and second 
attempt combined). In fact, absence of DVM course enrollments (program avoidance) 
was twice as big a problem (27 percent) and the single most important reason what 
students tended to lack math skills credentials, followed by failure to take the DVE 
placement exam in the first place ( assessment avoidance — 24 percent). Among DVM 
course-takers, poor grades did account for largest share of non-completers (29 percent), 
but final course withdrawal (program dropping ) was almost as telling a factor (25 
percent), and even stopping one’s program after receiving an advancing grade was a 
discemable trend (9 percent). For that matter, most course grade-related incompletes 
could be considered decisional rather than behavioral. For the definitional purest, only 
formal program failure (non-passing grade in the final course repeat attempt) would 
constitute genuine flunking of the program (just 6 percent of DVE course-takers); the 
others with non-passing grades but not making a second attempt to pass (23 percent) 
could be said to have opted to stop their programs before completion. 

Overall Remedial Effects 

The next step in the research plan called for investigating general remedial process 
effects on student progress, a tricky proposition since this involves somehow 
summarizing the degree track effects found in Table 3 across all three remedial areas. We 
took two related tacks to achieve this, the results of both shown in Table 4 below. In the 
first, we reconstituted the three developmental area career path variable of Table 3 into a 
set of discrete dummy variables representing any instance in a student’s career across all 
three developmental areas of a particular (e.g., any incidence of program evasion). Such 
indicators collectively examined provide a useful level of insight into cross-area relative 
importance of types of student decisions and behaviors for spoiling access to degree- 
culminating credit courses, overlapping case membership (a single student may exhibit 
up to three different any-instance develbpment paths) blurs interpretation. The second 
and more potent approach was to trick the many any-instance indicators into a single 
multi-category measure we call the Preclusion Cascade. The trick was accomplished by 
the employment of “trumping” rules. The Cascade assigns a degree-track precluded 
student to one and only one preclusion incident category, according to chronological or 
logical precedence. To illustrate, an occurrence of area assessment avoidance in a 
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student’s overall development career “trumps” any manifestation of program evasion or 
termination in other areas since it comes at the very start of that career. Similarly, a 
student who may have both withdrawn from a final course in one area and formally failed 
his final course in another would be assigned to the latter, unavoidable category. This 
procedure not only eliminates messy case overlaps but produces a variable of the discrete 
preclusion incidence categories which arrange themselves naturally by intervention 
priorities. Table 4's Preclusion Cascade percentage distributions tend strongly to 
underline our main individual area finding — that incomplete basic skills remediation is 
primarily a function of non-scholastic factors. The pattern is very clear: 23 percent of the 
cohort (37 percent of development incomplete students) left the degree track at the 
placement testing point of the remedial process by avoiding a skills assessment, another 
cohort 16 percent (26 percent of the non-completers) departed at the program start point 
by failing to enroll in required area courses. 



Table 4. Cross-Develo£mental Program Main Effects (Percentages^ 





The Preclusion 
“Cascade” 
Column % 


Any Instance 
Overlapping 
Categories 
% of Sample 




Cohort 

(2,094) 


Remedial 

(1,571) 


Cohort 

(2,094) 


Incomplete 

(1,297) 


Non-Developmental 


25.0 


' r '‘ 




", * f’ ' 


Developmental Required 


75.0 


100.0 




Completed All 


13.1 


17.4 




Non- Completers 


61.9 


82.6 


; V;£ 








Incomplete 

(1,297)* 


* *’ ‘ v.- : ■ ■ •" 


Total Assessment Avoidance 


6.8 


11.0 


6.8 


11.0 


1 or 2 Skipped Tests 


15.9 


25.6 


15.9 


25.6 


Program Evasion (No Courses) 


16.5 


26.7 


19.7 


31.8 


Formal Failure (2nd Attempt NP) 


1.3 


2.2 


1.4 


2.3 


1 st Non-Pass Grade/Drop 


7.7 


12.5 


12.5 


20.2 


Course Withdrawal/Drop 


7.1 


11.4 


11.0 


17.8 


Advancing Grade/Stop 


2.5 


4.1 


5.5 


8.9 



^’Incomplete” sub-column percentages sum to 100. 



Thus, almost two fifth of cohort members (over three-fifths of non -completer) lost all 
chance of graduating in a way that had nothing to do with what happens in developmental 
classrooms. The remainder of the non-completers had undergone all three skills 
assessments and entered all of their required programs, but many spoiled their graduation 
chances by failing to repeat a course in which they earned a non-passing grade (cohort 7 
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percent, non-completers 12 percent) and similar proportions effectively withdrew from 
the degree track by withdrawing from a last developmental course. Given all of the 
above, only 1 percent of the cohort (2 percent of non-completers) went off-track 
exclusively because of failed scholastic effort. 

Finally, at some point it occurred to us that much of this seemingly rampant 
developmental avoidance and withdrawal behavior might be spurious, an artifact of 
simple first term college attrition. So, we re-constructed the Preclusion Cascade to 
include a Termi dropout effect. The result was that first term attrition jump straight to the 
top of the list of developmental non-completion explanators (cohort 20 percent, non- 
completers 22 percent). However, although developmental avoidance and withdrawal 
effects were attenuated, they remained robust. For example, the proportion of students 
failing to complete due to program evasion dropped from 20 percent of the cohort to still 
strong 10 percent. Thus, we were able to conclude that — yes — first term attrition was an 
important source of remediation non-completion in our cohort — but, no — it far from 
explained away the sort of developmental evasion and retreat we were discovering. 

Student Developmental Career Clusters 

In the final phase of our research, we decided to drop the narrow focus on degree 
track preclusion. We wanted to search more freely for cohort developmental career 
patterns, using a broader set of remedial behavior variables and a methodology capable of 
bringing coherence to a wider range of cross-area remediation effects. Furthermore, this 
time we wished gain insight into what makes for successful as well as unsuccessful 
careers. 

Our research plan called for a k-means cluster analysis of cohort student requiring 
remediation in at least one developmental area (n= 1,094), using data representing not 
only their cross-area career paths (dummy variables derived from the three master 
developmental career variables), but also remediation need (e.g., number of required 
programs, level of program placement), program effort (e.g., number of courses taken per 
courses required, incidents of course repeats, major term duration of course-taking), and 
first term history (e.g., extent of credit course-taking and credit earning, first term 
attrition). The cluster analysis was stratified by degree of remedial success (all required 
programs completed, some completed, and none completed) for maximum clarity in the 
discernment of developmental career patterns tending toward positive or negative 
outcomes. Different analysis solutions within each outcome stratum (between two and 
five clusters) were generated and examined for levels of homogeneity and 
interpretability. The solutions finally accepted yielded a cross-strata set of nine clusters, 
which in Table 5 11 are named and briefly described by key statistics used in their 
derivation. 



11 Table 5 referenced in this article may be obtained by contacting the author. 
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Space prohibits an elaboration of the nature of each individual cluster, but the 
following general observations bearing on developmental careers and remediation 
success can be made: 

Full Completion Clusters. Two clusters emerged in this stratum, one composed 
mostly of students needing only, and easily completing, brush-up in single skill areas, and 
another characterized by multi-skills deficiencies who had only moderate trouble with 
their DVE and DVR programs, but had to fight their way to victory in DVM. Although 
the Brush-Up cluster was twice as populous as the Math+ Champion group, it was 
somewhat mollifying to discover that program success at PGCC was not completely 
confined to those developmental students least needing remediation. Furthermore, an 
analysis of the academic outcome for these two clusters revealed similar above-average 
levels of degree and transfer attainment, suggesting that the oft-observed phenomenon of 
the remediated super-student includes the marginally skills-deficient but the hard cases as 
well. 

Partial Completion Clusters. These three are the “heartbreak” clusters, so near to and 
yet so far from entering the degree track. Both the Multiple Area Stragglers and All-But- 
Math Fighters made major efforts to overcome cross-area skills deficits but ultimately fell 
short, the latter held back from victory only by mathematical inaptitude. The third and 
majority cluster in this stratum also was balked the mainly in math remediation, but here 
a sort of failure of nerve rather than defeat in battle seemed to be involved The Math 
Dodgers manifested high levels of math program evasion, and DVM programs that were 
start usually terminated in course withdrawal. 

No Completion Clusters. In this largest of strata (including over half of all program 
non-completers), four clusters emerged. Two small groups (the Math Defeated and 3R 
Lost Causers) battled heroically but to zero effect. Energetic effort by the former could 
not overcome a single area deficiency in math, and the latter, most cross-area deficient of 
any cluster, suffered a general rout. But dominating the stratum were two large clusters 
whose defeat was mostly self-inflicted. The large plurality (near 40 percent) of the 
multiply skills deficient 3R Dropouts engaged in program evasion and the remainder 
tended to terminate their programs after a single course. The final product of the stratum 
in cluster sort was the Math Dodger group, the most populous of any and embracing 
three out of ten of all developmental non-completers in the cohort. These here turned out 
to be math-only remedial students predominantly. Even so, the prospect of undergoing 
remediation in just this one area proved to daunting for them, 70 percent dodging all 
DVM course-taking. 

Perhaps a quick review of the lessons taught by these cluster analysis patterns may 
also stand as a summary of the key findings of the entire study. If the way Prince 
George’s Community College’s remedial education process functions is any example, 
then: 
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The complexity of the remedial process allows for the emergence of many 
developmental career types and multiple paths to both happy and unhappy 
conclusions. This suggests that for maximum effectiveness in remediation, 
developmental programs should to be particularized to fit the diverse needs, 
abilities and prospects of each type. 

Motivation matters, but is not decisive. For the multiply skill deficient, program 
effort and dedication is a necessary but not sufficient condition for a successful 
remedial career. Several developmental career types struggle heartily but futilely 
toward the completion of their remediation. These should be the prime targets of 
student support services generally and special intervention programs in particular.. 

Poor developmental career decision-making accounts for more remediation non- 
completion than does poor developmental course performance. The maze-like 
complexity of the remediation process includes many dead-end corridors off the 
marked path, and the indirect nature of degree track preclusion by credit course 
skill prerequisite encourage the false impression among students that 
developmental evasion and withdrawal are viable academic options. 

The developmental maze proves very difficult for student to thread properly. 
Advisement services should be appropriately enlarged and energized to mitigate 
the reluctance, confusion, frustration and panicked search for short-cuts that 
developmental education, like physical labyrinths, inherently foster. 

Last but far from least, the specific character of the remediation process at a 
postsecondary institution may make its own, independent mark on that school’s 
academic outcomes. In PGCC’s case, the remediation process functioned to 
effectively shaved the actually graduateable student body to about a third of the 
numbers of degree-seekers appearing in its institutional reporting. That is power 
indeed. 
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The growth of the outcomes assessment movement in higher education has been 
dramatic in the past decade. In the public sector, colleges and universities have come 
under increasing pressure from their constituencies to demonstrate their accountability, 
effectiveness, and efficiency in measurable terms. As a result, many institutions, 
especially public college/university systems, have adopted some kind of performance 
indicator systems with simple and quantifiable measures (Borden & Banta, 1994; Cheng 
& Voorhees, 1996). In the private sector, though many institutions, especially the elite 
ones, still enjoy the favorable ratings by US News and World Report and other agencies 
using “reputational” and “resources” approaches (Jacobi, Astin, and Ayala, 1987), the 
general sense of crisis is deepening. The public, students, and their parents demand to 
know whether private, elite institutions are delivering what they promised, and whether 
they are doing so in a cost-effective, high-quality way (Upcraft & Schuh, 1996, p. 8). 

While these myriad pressures have prompted college administrators to scramble for 
assessment models that fit their own institutions, there is also mounting evidence showing 
that the fundamental question of outcomes assessment, i.e., What is to be assessed? is 
often overlooked. The list of performance indicators compiled by Bottrill and Borden 
(1994) from various sources reveals the general tendency of institutions in moving 
toward a system of indicators that are quantifiable, easy to capture, and usually having 
the appearance of objectivity. Student test scores on aptitude, GPA’s, 
retention/persistence/graduation rates, etc., are among the most popular indicators 
adopted. While all these indicators do indeed measure certain aspects of an institution’s 
effectiveness, the biggest drawback, however, lies in their inability to provide meaningful 
information on students’ intellectual and personal development as the outcomes of their 
collegiate experience. Consequently, institutions adopting performance indicators 
typically find it difficult to include any indicators that can reliably measure the less 
tangible aspects of students’ collegiate experience. 

Literature Review 

In their 1987 ASHE-ERIC Higher Education Report Jacobi, Astin, and Ayala (1987) 
proposed an alternative conception of “talent development” to counter the popular 
definitions of excellence using the reputational and resource approaches. Jacobi, Astin, 
and Ayala (1987) believe that “a high quality institution is one that maximizes the 
intellectual and personal development of its students” (p. iv). 
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This report was among a considerable number of studies carried out to explore 
different taxonomies of the outcomes of college. Other influential studies include: Astin, 
1973; Brown & DeCoster, 1982; Chickering & Gamson, 1987; Ewell, 1984, 1985a, 
1985b, 1988; Hanson, 1982; Kur, Pace & Vesper, 1997; Kur, Hu & Vesper, 2000; 
Lenning, Lee, Micek, & Service, 1977; and Pascarella and Terenzini (1991). The 
importance of the research in this area, according to Jacobi, Astin, and Ayala (1987), is to 
provide a useful “menu from which researchers and practitioners may select the items of 
greatest importance to measure and track” (p. 19). 

Of the frequently cited typologies, Astin ’s (1974, 1977) provides a three-dimensional 
taxonomic system: by type of outcomes: cognitive vs. affective; by type of data: 
psychological vs. behavioral; and by time: short-term vs. long-term. To a large extent 
Astin ’s taxonomy is more of a framework for outcomes than actual outcome categories, 
as they are the case in Lenning (1977, 1980) and Bowen (1980). Mentkowski & 
Doherty’s (1983) typology is more practically-oriented, developed by faculty and 
administrators at Alvemo College to implement an outcome-centered liberal arts 
program. 

In the national scene, a number of attempts have been made in recent years to convert 
students’ behaviors, cognitions, and attitudes enhanced through collegiate experiences 
into outcome indicators (National Center for Education Statistics (NCES), 1991 ; National 
Education Goals Panel, 1992; National Center for Higher Education Management 
Systems (NCHEMS), 1994). It is of no surprise that researchers or research groups differ 
considerably among themselves in their developed categories or taxonomies of outcome 
measures. However, common to most of these attempts is that the assessment of student 
behaviors, cognitions, and attitudes has to rely heavily on subjective measures using 
student self-perceived intellectual, social, and personal gains. “For some outcomes, 
student reports may be the only source of useful data” (Kur, Pace & Vesper, 1997). The 
College Student Experience Questionnaire (CSEQ) (Pace, 1979) and the College Student 
Survey (Higher Education Research Institute, 1989) are among the most widely used 
survey instruments that include items of student self-reported gains in college. The 
results of research using student self-reports of growth are in general consistent with 
research using other measures of collegiate achievement (Anaya, 1999; Pace, 1985; Pike, 
1995). 



Research Questions 

In the ideal world of assessment, an institution is supposed to go through a cycle from 
setting missions, goals, and objectives, to developing instruments to assess the 
effectiveness of institutional performance as related to the goals, and finally to making 
improvements using the assessment results (Moxley, 1999). However, in the real world, 
few institutions find themselves completing such a perfect cycle due to all kinds of 
constraints. For instance, limited by time, expertise, and the lengthy testing cycle, an 
institution can hardly afford to locally develop a valid and reliable instrument that 
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assesses exactly what the institutional goals or missions call for. Therefore, a common 
alternative is to adopt a commercial survey instrument or to join a research consortium 
and use a consortium-developed survey instrument. 

Once an institution adopts such an externally-developed survey instrument to assess 
student collegiate experience, it’s not unusual that they find themselves caught in a 
dilemma: on the one hand, they have all these wonderful theories, taxonomies, or 
typologies that they want to use to assess their students’ collegiate experience; on the 
other hand, the survey instrument they adopt is either not specific enough to address 
certain unique institutional experiences, or it simply contains too many items which not 
only blur the focus of institutional assessment goals but also make the results hard to 
interpret. 

With all the existing outcome taxonomies as the research framework, the purpose of 
this study, therefore, is two fold: 1) to analyze an array of questions on student self- 
perceived gains in college using an externally-developed survey instrument, aiming at 
developing several comprehensive Student Self-Perceived Gain Scales (SSGS) to support 
an institution’s assessment of student collegiate experience, and 2) to test the utility of the 
developed SPEGs and their association with various characteristics of a student body in a 
private, highly selective institutional environment. 

Methods 

The data used in this study is from a senior survey of graduating classes of 1997, 

1998, and 1999 at a private, urban, and highly selective research university. Because the 
institution requires that the graduating seniors complete the survey before picking up 
their graduation tickets, the response rates were close to 100%. The total number of cases 
included in 1997, 1998, and 1999 files is 1,057, 1,104, and 1,103 respectively. The 
respondents were graduates of two undergraduate colleges: the college of arts and 
sciences (A&S) and the college of engineering (ENGR). The survey instrument was 
designed by a consortium of highly selective institutions to assess different aspects of 
their students’ experience in college, and the questions range from graduates’ future 
plans, evaluation of undergraduate experience, financing of undergraduate education, 
college activities, and demographic background. There are twenty-four questions in the 
survey asking about students’ self-perceived gains. 

An exploratory factor analysis of the twenty-four items concerning student self- 
perceived gains was conducted using the 1997 survey data. Principal component analysis 
with varimax rotation was utilized for interpretability. Since the purpose of the analysis 
was not data reduction but creation of meaningful scale variables using all the available 
data, no item was eliminated because of low factor loading. Based on the results of the 
factor analysis, composite scales were constructed and the same items were grouped for 
all three years’ data respectively. Existing taxonomies were used as the frame of 
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reference to discern the most meaningful scales in describing students’ self-perceived 
gains in college (SSPGs). 

Reliability analyses were then conducted for all the scale variables to determine the 
appropriateness of items used to for grouping. Correlation and alpha indexes of both 
scales and individual items were examined and compared across three years to check the 
consistency and stability of the developed scales. 

After the SSPGs are constructed, two sets of independent variables were extracted 
from the survey data to test the utility of SSPGs. The first set of variables includes 
student demographic characteristics: sex, ethnicity, citizenship, family income, and 
parents’ highest educational level. The second set has to do with three important aspects 
of student college experience: GPA, the major field of their degree, and the overall 
satisfaction level of their undergraduate experience (l=very dissatisfied; 5=very 
satisfied). 

The tests of utility of developed SSPGs followed a two-step process. First, with each 
SSPG considered separately, multiple regression procedures were performed to discern 
the associations between independent variables and each SSPG. Second, with all the 
SSPGs considered simultaneously, multivariate analysis of variance (MANOVA) 
procedures were conducted to examine the differences of the two colleges (A&S and 
ENGR) and three levels of satisfaction (l=dissatisfied; 2=ambivalent; 3=satisfied) as 
independent variables on the five SSPG scales. The rationale behind these tests were: 1) 
a SSPG is a good measure of student gains if it displays some level of consistency in the 
way it interacts with independent variables across different years’ data; and 2) the SSPGs 
are good measures of student gains if it has disparate impact on students who were 
affiliated with different colleges and reported different levels of satisfaction with their 
college experience. 

It should be noted that these procedures were used for multiple purposes, not simply 
statistical inference. As a matter of fact, since the entire populations of the three classes 
were used for the analyses, statistical inferences are barely necessary. The inferential 
results would make sense when the data were supposed to constitute a random sample. In 
research practice, nonetheless, tests of significance were often used to analyze non- 
random data, with the results pointing to the presence of a relatively considerable effect. 
The inferential results included in this study should only be interpreted in such a manner 
(Chen, 1998; Chen & Cheng, 1999). 



Results 

Table 1 shows the rotated factor structure of the five-factor solution. A content 
analysis yielded the following grouping of the scale variables: 1) Practical competence 
(Bowen’s (1980) term); 2) Human characteristics (Lenning’s (1977, 1980) term); 3) 
Leadership competence; 4) Academic ability; and 5) Foreign language skills. Note that 
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the only items with factor loading lower that 0.5 are “Function independently, without 
supervision” in factor 1 and “Understand myself: abilities, interests, limitations, 
personality” in factor 2. These two items were nonetheless retained for their meaningful 
contribution to the respective scales. 

The application of factor analysis results using the 1997 data to the data of following 
years yielded stable and consistent scale variables. The ranges of the alpha values from 
the reliability analyses of the data of 1997 to 1999 are: 0.85-0.87 for scale 1; 0.83-0.86, 
scale 2; 0.85-0.86, scale 3; 0.61-0.63, scale 4. Scale 5 is a single item. 

Table 2 is a summary of the results of multiple regression analyses conducted to 
examine the associations of independent variables with each of the five self-perceived 
gain scales. Apparently, the level of satisfaction with undergraduate education is the 
factor most closely associated with students’ self perceived gains in college. Student 
major also seems to play an important role in their self-perceptions. While natural 
science and engineering majors perceived having higher gains in academic ability than 
humanities and social science majors, engineering majors were less confident about their 
gains in human characteristics and foreign language skills than their counterparts in other 
majors. In general, the self-perceived gains of the graduates are less influenced by their 
demographic and socioeconomic background than by college-related variables. 

The MANOVA procedures for students’ college affiliation and satisfaction on the five 
SSPGs for all three years (Table 3) were statistically significant by the Wilks’ Lambda 
criteria (F=7.76, df=5/10, p<.01 for 1997; F=9.01, df=5/10, p<.01 for 1998; F=12.04, 
df=5/10, pc.Ol for 1999). Inspection of the univariate F-ratios reveals statistically 
significant differences among the three satisfaction levels on four of the five SSPGs, with 
the only exception on foreign language skills for the 1997 and 1999 models. Graduates 
of the two colleges also show statistically different self-perceptions on four of the five 
SSPGs, with the exception on practical competence. However, none of the 
college/satisfaction interactions is statistically significant. Further analyses of means 
broken down by college and satisfaction level confirmed that, despite the differences in 
level of satisfaction, students from both colleges show the same pattern of self- 
perceptions on all the five SSPGs: the higher the satisfaction level, the better they felt 
about their gains in the five areas. 

One noteworthy pattern emerges from examination of both the regression and the 
MANOVA results for all three years’ data: the perceptions of students from these three 
cohorts were very consistent. For instance, females consistently showed higher self- 
perceived gains in human characteristics than their male counterparts (betas are .06, .09, 
and .09 for 1997, 1998, and 1999 in Table 2); humanities students tended to report higher 
gains in foreign language skills than those from other majors (betas are .07, .10, and .15 
for 1997, 1998, and 1999 in Table 2); and no statistical significance existed between 
A&S and ENGR students in their self-perceived gains in practical competence. 
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Summary and Discussion 



The analyses of the twenty-four questions regarding student self-perceived gains from 
an externally-developed senior survey yielded five outcome scales: 1) Practical 
competence; 2) Human characteristics; 3) Leadership competence; 4) Academic ability; 
and 5) Foreign language skills. Analyses show that the level of student satisfaction with 
undergraduate education is closely associated with their self-perceived gains in college. 
Student major also seems to play an important role in their self-perceptions. In general, 
graduating seniors’ self-perceived gains are less influenced by their demographic and 
socioeconomic background than by college-related variables. Given the consistency of 
SSPGs over a three-year period and their disparate impact on students with different 
characteristics, we can comfortably conclude that the students’ perceptions of their 
collegiate experience in this particular institution are well represented in the five SSPGs 
derived from self reports. 

In the past decade the idea of assessing “how much students learn or improve or grow 
in school or in college, as well as how they stand at graduation” (Belcher, 1987) has been 
gaining momentum over the traditional “reputational” and “resources” approaches. This 
study is a demonstration of how this new approach can work even if an institution has 
already committed to using an externally-developed survey to assess student collegiate 
experience. The five student self-perceived gain scales (SSPGs) derived from the 
graduating senior survey have not only presented the student version of the outcome 
measures of their collegiate experience, but also are comprehensive and meaningful to an 
institution that has a long tradition of emphasizing the breadth of learning through general 
education and community services. The usefulness of this study is that any institution 
can follow the methodology demonstrated in this study and derive its own outcome 
measures of student collegiate experience using whatever student self-reports they have 
chosen. 

However, being able to form outcome measures does not necessarily mean that an 
institution has found the answer to the critical questions of what is excellence in higher 
education and how it can be attained and assessed. The lesson we learned in this study is 
that the process of searching for outcome measures itself is an institutional “soul- 
searching” process, in which the college community has to revisit and/or redefine its 
institutional missions and goals constantly. The fact that so many taxonomies can be 
used for assessing college outcomes clearly shows that there can be as many ways of 
defining excellence in higher education. The ultimate goal of student assessment, 
however, should be to use the results of the assessment to readjust the existing mission 
and goals, and thus to provide a better institutional environment for student learning and 
growth. 
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Table 2. Regression Beta Weights for the 5 Scales with Student Characteristics. 
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Note: All the beta weights listed in the table are significant at the .05 level (pc. 05). 
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INSTITUTIONAL RESEARCHERS: CHALLENGES, 
RESOURCES AND OPPORTUNITIES 



Anne Marie Delaney 
Director of Institutional Research 
Babson College 



Purpose. This paper presents the results of a study that investigated challenges 
institutional researchers encounter in their career; resources for coping with these 
challenges; and the impact of these challenges on job quality and on engagement in 
policy. The major research questions addressed in this study are: 

• What are the primary professional challenges institutional researchers encounter? 

• How do these challenges vary by level of position and use of resources? 

• To what extent do level of position, challenges and resources predict job quality? 

• What impact do challenges have on institutional researchers' engagement in policy? 

• How do job quality, level of position, challenges and resources predict 
involvement in policy? 

The goal of this research is not only to identify and understand the problems, but also to 
propose creative strategies to meet these challenges and thus enhance institutional 
researchers' professional status and effectiveness. 

In the context of this study, professional challenges encompass immediate concerns 
as well as difficulties experienced during the course of one's career. Three major areas 
addressed include: concerns about one's current job; difficulty in securing support for 
one's values and work; and pressure to compromise to meet career demands. 

Review of the Literature. During the last three decades, researchers have 
investigated the problems and challenges institutional researchers encounter in their 
professional practice. Gubasta (1976) defined problems facing college decision makers 
and increasing information needs of external agency representatives as sources of 
conflicting pressure on institutional researchers. Storrar (1981) identified role conflict to 
be a source of stress for institutional researchers. She found that while institutional 
researchers perceived their actual roles as high on political responsiveness and political 
advocacy, they preferred roles of policy advocacy and low political responsiveness. 
Sanford (1983) cited little extrinsic recognition for the work and the need to work with a 
number of other persons and offices without having direct control as primary sources of 
stress for institutional researchers. Huntington and Clagett (1991) reported insufficient 
staff; excessive workload; lack of access to quality information and decision-makers; and 
inadequate training of staff as problems most frequently experienced by institutional 
researchers. 
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Matier, Sidle and Hurst (1995) offer ideas for meeting such challenges. They 
recommend that institutional researchers exercise leadership in defining their work and 
expand their sphere of influence by assuming roles as information architects, change 
agents, and consultants of choice within their respective institutions. Hurst, Matier and 
Sidle (1998) also propose that institutional researchers serve as facilitators of the learning 
process as a way of enhancing the role of institutional research and that institutional 
research play a key role in promoting the success of teams to ensure that decisions are 
grounded in the support of institutional constituents. Such initiatives may strengthen 
institutional researchers' ability to meet the challenges of demanding workloads and 
expand the possibilities for decision-making influence and professional advancement. 

Data Source. Data for this study are based on results from a mailed survey sent to 
304 institutional researchers in the Northeast; 221 returned completed surveys yielding a 
response rate of 73 percent. The respondent group reflects the demographic, educational 
and professional diversity of the institutional research profession. Of the 221 
respondents, 41 percent are male and 59 percent are female; 40 percent possess a 
doctorate; 42 percent have a master’s degree; and 18 percent hold a bachelor’s degree. 
Respondents represent a range of professional positions. Eleven percent hold titles at the 
level of dean to vice-president; 50 percent are directors; 10 percent are associates; 16 
percent are analysts, coordinators or mangers; and 13 percent are assistants or research 
and technical specialists. 

Analytical Techniques. Analyses were conducted with individual survey items and 
computed scales. The scales represent the following constructs; engagement in policy, 
job quality and professional challenges. Bivariate techniques - correlation, chi-square, 
t-test, and analysis of variance - examined the relationships between level of position, 
resources and challenges. Path analysis assessed the direct and indirect effects of level of 
position, resources and challenges on job. quality and on engagement in policy. 

Scale Development . Factor analyses were conducted to establish construct validation, 
that is, to identify the unidimensional or multidimensional constructs underlying the 
items related to professional challenges, job quality, and engagement in policy. Common 
factor analysis or the principal axis factor method was employed. This method was 
chosen since it assumes that the factors are correlated. 

Results from factor analyses indicated which individual items were correlated with 
each other and what underlying dimensions were represented in the data. Factors were 
selected that explained a substantial amount of variance and included at least two or more 
items. Scales were then created by combining similar items into one measure. 

Generally, items with factor loadings of .5 or higher on a particular factor were chosen to 
be included in a scale. Prior to using the scales in the analysis, alpha reliability 
coefficients were computed to determine the internal consistency of the scales. 

Table 1 presents the names, statistical properties, and correlations among these scales. 
Items comprising these scales are presented in Appendix A. The reliability of these 



scales is very high with coefficients ranging from .80 to .90. As reflected in the mean 
scale scores, the most prevalent challenge among institutional researchers involves 
experiencing overwhelming demands in their current jobs, followed by managing conflict 
between work and personal/family needs, coping with limited opportunity and dealing 
with threats to quality standards. The moderately high means on engagement in policy 
and job quality suggest that many institutional researchers are involved in policy and 
have a quality work experience. 



Table 1 

A. Statistical Properties of the Scales 













Range of 






St. 




No. of 


Responses 




Mean 


Dev. Reliability Items 


Low-High 


Professional Challenge Scales 












a. Experiencing Overwhelming Demands 


3.25 


1.07 


.87 


3 


1-5 


b. Managing Conflict between Work and Family 


2.56 


.95 


.83 


3 


1-5 


c. Coping with Limited Opportunity 


2.48 


1.00 


.89 


6 


1-5 


d. Dealing with Threats to Quality Standards 


2.07 


.77 


.80 


2 


1-5 


Work Experience Scales 












e. Job Quality 


3.72 


.70 


.86 


12 


1-5 


f. Engagement in Policy 


3.24 


.82 


.89 


10 


1-5 


B. Correlation among the Scales 




a b 


c 


d 


e 


f 


a. Experiencing Overwhelming Demands 


.56*** - 


_ 




. 21 ** 


b. Managing Conflict between Work and Family 




- 


.17* 




^ 5 *** 


c. Coping with Limited Opportunity 






44 *** 


_ 47 *** 


-.26*** 


d. Dealing with Threats to Quality Standards 








_ 43 *** 


_ 23*** 


e. Job Quality 










*** 


f. Engagement in Policy 













* p < .05; ** p < .01; *** p < .001 
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As shown in Table 1, correlation analyses results identified statistically significant 
correlations among some of the scales. A strong positive correlation exists between 
experiencing overwhelming demands and managing conflict between work and family. 
A moderate, significant correlation also exists between coping with limited opportunity 
and dealing with threats to quality standards. Engagement in policy is positively 
correlated with experiencing overwhelming demands and managing conflict between 
work and family. Job quality and engagement in policy are negatively correlated with 
coping with limited opportunity and dealing with threats to quality standards. Finally, a 
strong, positive correlation exists between job quality and engagement in policy. 

Results 



Frequency of Challenges 

This section on the nature and frequency of challenges among institutional 
researchers presents results from analyses based on individual survey items and 
computed scales. 

Concerns about Current Job . Figure 1 identifies the top six specific aspects of their 
current job that institutional researchers describe as 'very much' of a concern. As shown, 
three of these concerns relate to work demands - having too much to do, the job is taking 
too much out of you, and stressful demands of the job. The other two frequently reported 
concerns relate to career advancement: having little chance for advancement and limited 
options for career development. 



Figure 1 

Institutional Researchers' Concerns about their Current Job 

Percent Reporting 'Very Much' 
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Challenges during Research Career . Figure 2 shows the percent who reported they 
experienced various challenges 'very much’ during their career. These challenges refer to 
obtaining support for one's values and standards; securing resources to conduct the work; 
and obtaining support in resolving conflicts and ethical issues. As shown, 24 percent 
report that producing quality work within time constraints has been 'very much' of a 
challenge. Between 13 and 15 percent also report the following issues have been 'very 
much' of a challenge during their career: receiving credit for work; finding opportunities 
to be heard; and attaining support for professional standards. These data identify 
potentially serious issues as these challenges threaten institutional researchers' 
professional status, job quality, and potential for advancement. 



Figure 2 

Institutional Researchers* Career Challenges 



Producing quality work within time 
constraints 

Receiving credit for the work you do 

Finding opportunities to make your voice 
heard 

Attaining support for your professional 
standards 

Obtaining necessary resources 
Receiving support for personal values 
Gaining support for your work 
Securing support with an ethical dilemma 
Resolving conflicts with superiors 



Percent Reporting 'Very Much' 

0% 5% 10% 15% 20% 25% 30% 
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Pressure to Compromise . Figure 3 identifies the top four compromises respondents 
indicated they 'frequently' or 'very often' felt they had to make for their career. As shown, 
institutional researchers most frequently cited pressures related to work demands and 
professional integrity. Some 25 percent cited working excessive overtime; 21 percent 
reported neglecting personal needs; 14 and 12 percent respectively reported allowing 
others to take credit for their work and performing work with inadequate training. 



Figure 3 

Pressures to Compromise Experienced by 
Institutional Researchers 



0% 



Work excessive 
overtime 

Neglect personal 
needs 

Allow others to take 
credit for your work 

Perform work with 
inadequate training 



Percent Reporting 'Frequently' or 'Very Often' 

5 % 10 % 15 % 20 % 25 % 30 % 




21% 



14 % 




12% 



Variation in Challenges 

Bivariate analyses were conducted to answer the question: How do professional 
challenges vary by level of position and use of resources? These analyses included 
t-tests, analysis of variance and the Student-Newman-Keuls post hoc test to determine 
where the significant differences occur among institutional researchers. These analyses 
were conducted with individual survey items and with computed scales. 

Level of Position and Professional Challenges . Results based on the individual 
survey items, revealed statistically significant differences between level of position and 
the following professional challenges that relate to work demands: the job is taking too 
much out of you (F = 3.28, p < .05); working excessive overtime (F = 6.08, p < .001); 
neglecting family responsibilities (F = 4.1 1, p < .01); and neglecting personal needs (F = 
3.47, p < .01). Further, the Student-Newman-Keuls post-hoc test results indicated that 
these challenges were significantly higher among institutional researchers holding the 
highest level positions from dean to vice president. 
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Level of current position was also significantly related to minimal opportunity to use 
one's intelligence (F = 2.72, p < .05); job monotony or lack of variety (F = 3.41, p <_.01; 
and pressure to lower one's standards (F = 2.59, p < .05). These challenges, which 
involve the intellectual quality and integrity of one's professional life, were generally 
highest among research analysts and associates. 

Scale level analyses revealed statistically significant differences between level of 
position and two challenge scales: experiencing overwhelming work demands 
(F = 3.19, p < .05) and managing conflict between work and personal/family needs 
(F = 6.35, p < .001). The means were highest among those holding positions from dean 
to vice president. According the Student-Newman-Keuls test results, the difference was 
statistically significant on managing conflict between work and personal/family needs. 

Resources and Professional Challenges . T test results documented the value of a 
mentor and a strong professional network in coping with professional challenges. Those 
who had a mentor were significantly less likely to report that the job was taking too much 
out of them (t = 2.25, p < .05) or that they were having difficulty in obtaining necessary 
resources for their work (t = 2.05, p < .05). Also, those who were part of a strong 
professional network were significantly less likely to report the following concerns about 
their present job: little chance for advancement (t = 1.97, p < .05); limited options for 
career development (t = 3.14, p < .01); minimal opportunity to use one's intelligence 
(t = 3.45, p < .001); inadequate opportunity to show creativity (t = 2.46, p < .05); and job 
monotony or lack of variety (t = 3.45, p < .001). 

Institutional researchers who report they are part of a strong professional network also 
report they are significantly less likely to experience pressure to make professional or 
ethical compromises, including to perform work with inadequate training 
(t = 4.40, p < .001); to present a false, less competent image ( t = 3.69, p < .001); to 
sacrifice quality (t = 2. 16, p < .05); or to treat others unfairly (t = 2.20, p < .05). 

Further analysis with the challenge scales identified a statistically significant 
relationship between having a mentor and coping with limited career opportunity 
(F = 4.13, p < .01). This challenge was highest among those who did not have a mentor 
and lowest among those who had both a male and female mentor. Those who reported 
they were part of a strong professional network also reported significantly less challenge 
in dealing with threats to quality standards (F = 2.52, p < .05). 

Path Analysis Technique . Path analysis was employed to answer the following 
questions. To what extent do level of position, challenges and resources predict job 
quality? How do job quality, level of position, challenges and resources predict 
involvement in policy? Technically, the path-analytic technique assessed the direct and 
indirect effects of a set of exogenous variables - level of position, challenges and 
resources - on an endogenous variable - job quality and the effects of all of the exogenous 
variables and job quality on engagement in policy. 
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Figure 4 shows the results visually in a path diagram. The lines indicate the pathways 
that had beta-weights greater than .10, with the specific beta-weight indicated for each 
pathway. Each path coefficient is the beta-weight for the precursor variable on the 
endogenous variable. In an attempt to control for practical significance, when the 
standardized regression coefficient (beta-weight) for a particular path was less than .10 
(Hackett, 1985), the path was dropped. 



Figure 4 

Path Diagram for Predicting Engagement in Policy 
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The calculations of the direct and indirect paths are presented in Table 2. This causal 
analysis decomposes the correlation between two variables into three components: direct, 
indirect, and spurious. The direct and indirect components are summed to the total true 
causal effects whereas the spurious component is due to unexplained factors and is 
obtained by subtracting the total effect from the bivariate correlation coefficient. The 
direct effects are the effects that come directly from the precursor variable in the 



dependent variable, without being mediated by other variables in the model. The indirect 
effects are the effects of the precursor variable as operating through or mediated by other 
variables on the dependent variable. For example, the zero order correlation between 
level of position and engagement in policy is .49. Path analysis documents that the direct 
and indirect effects respectively are .26 and .17. The total effect is .43 and the spurious 
effect is .49 - .43, or .06. 



Table 2 

Path Analysis Results: Breakdown of Direct and Indirect Effects on Engagement in Policy 

Effects 



Path 


Bivariate r 


Indirect 


Direct 


Total 


Spurious 


Professional Network 


.29 


.06 




.06 


.23 


Threats to Quality 


-.23 


-.15 


- 


-.15 


-.08 


Conflict between Work and Family 


.25 


.06 


- 


.06 


.19 


Limited Opportunity 


-.26 


-.09 


- 


-.09 


-.17 


Mentor 


.16 


.11 


- 


.11 


.05 


Level of Position 


.49 


.17 


.26 


.43 


.06 


Job Quality 


.61 


~ 


.49 


.49 


.12 



Correlations . As illustrated in Table 2, statistically significant correlations were 
found between engagement in policy and each of the exogenous variables: level of 
position ( r = .49, p <_.001), followed by having a strong professional network ( r = .29, p 
< .001), and managing conflict between work and family ( r = .25, p < .001). Having a 
mentor is also positively related to engagement in policy ( r = .16, p < .05). In contrast, 
two of the professional challenges - coping with limited opportunity (r = -.26, p < .001) 
and dealing with threats to quality standards ( r = -.23, p < .001) are negatively correlated 
with policy engagement. Job quality has the strongest positive correlation with 
engagement in policy (r=.61, p < .001). 

Path Analysis Results . As reflected in the path coefficients, four of the six exogenous 
variables have a positive, direct effect on job quality. In order of magnitude, these 
variables are: level of position (.34), mentor (.22), professional network (.12) and 
conflict between work and family (.12). In contrast, two variables: dealing with threats 
to quality standards ( - .31) and coping with limited opportunity ( - .18) have negative 
effects on job quality. As indicated by the R 2 of .47, these variables explain 47 percent 
of the variance in job quality. 

All of the exogenous variables also have indirect effects, through job quality, on 
engagement in policy. These indirect effects range from - .15 for dealing with threats to 



quality to +. 17 for level of position. Further, level of position is the only exogenous 
variable that has a direct effect (.26) on engagement in policy. The R 2 of .43 
demonstrates that the direct effects of job quality and the direct and indirect effects of the 
exogenous variables explain 43 percent of the variance in engagement in policy. 

Discussion 

Results from this research confirm findings from previous studies that addressed 
challenges institutional researchers encounter in their career. In this study, approximately 
two-fifths identified having too much to do as very much of a concern in their current 
job. Close to one-quarter also reported that producing quality work within time 
constraints was very much of a problem in their career. In an earlier study, Huntington 
and Clagett (1991) also reported excessive workload as one of the problems most 
frequently experienced by institutional researchers. 

Recognition for the work accomplished is also a problem for a substantial number of 
institutional researchers. In this study, 15 percent reported receiving credit for work as 
very much of a challenge and 14 percent reported they frequently or very often felt they 
had to allow others to take credit for their work. These results involve an ethical issue 
regarding attributing appropriate credit to the person who accomplishes the work. In a 
previous study, Sanford (1983) identified little extrinsic recognition for the work as a 
primary source of stress for institutional researchers. 

This study documents clearly that those who have a mentor or are part of a strong 
professional network have higher job quality and are significantly less likely to 
experience many potential sources of stress on their job: such as, minimal opportunity to 
use one's intelligence, inadequate opportunity to show creativity, job monotony, or little 
chance for advancement. These positive effects of mentors and professional networks 
highlight the value of professional relationships. In this sense, the study supports the 
recommendation of Hurst, Matier and Sidle (1998) that institutional researchers promote a 
team approach as a way of enhancing effectiveness. 

Recommendations 

As noted previously, the goal of this research has been not only to identify and 
understand the challenges institutional researchers face but also to propose creative 
strategies to meet these challenges and thus enhance institutional researchers' 
professional status and effectiveness. Based on the study findings, the following 
recommendations are offered to achieve this goal. 

• The institutional research profession should promote strong mentoring relationships. 
Professional associations should provide the structures for developing mentoring 
relationships. Institutional research directors and university administrators should 
provide resources and create opportunities to support mentoring relationships for 
institutional researchers, particularly those who are new to the profession. 



• Institutional researchers should actively participate in professional associations and 
seek out colleagues for advice and support on a continuing basis. Regional and national 
associations should place a high priority on using the organizations to strengthen 
professional networks for new and experienced researchers. In additional to annual 
meetings, the associations should seek new ways to support networks during the year. 

• The institutional research profession should advocate that institutional researchers' 
jobs be structured with a high level of independence, intellectual vigor and professional 
integrity. Director's positions should be characterized by flexibility in establishing work 
priorities; authority in setting research agenda; freedom in deciding how work is 
accomplished and authority required to get the work done. All positions, especially 
research associate and analyst positions, should offer opportunities for intellectual 
stimulation, creativity and career advancement. 




49 



5 ? 



Appendix A 

Questionnaire Items Comprising the Professional Challenges Scale 



Experiencing Overwhelming Demands (r=.89) * 

When you think about your current job, how much, if at all, 
are the following items a concern for you? 

a. The job is taking too much out of you 

b. Having too much to do 

c. Stressful demands of the job 

Coping with Limited Opportunity (r=.89) * 

When you think of your current job, how much, if at all, 
are the following items a concern for you? 

a. Having little chance for advancement 

b. Lack of recognition 

c. Limited options for career development 

d. Minimal opportunity to use your intelligence 

e. Inadequate opportunity to show creativity 

f. The job's monotony or lack of variety 

Managing Conflict between Work and Family (r=.83) ** 

Do you feel you have had to make any of the following 
compromises to sustain your career? 

a. Work excessive overtime 

b. Neglect family responsibilities 

c. Neglect personal needs 

Dealing with Threats to Quality Standards (r=.80) * 

Do you feel you have had to make any of the following 
compromises to sustain your career? 

a. Lower your standards 

b. Sacrifice quality 



* Response Scale: 1 'Not at AH' to 5 'Very Much' 

** Response Scale: 1 'Never' to 5 'Very Often' 



Appendix A 

Questionnaire Items Comprising the Professional Challenges Scale 



Engagement in Policy (a= 89) *** 

Indicate the extent to which the following statements 
describe your role or the use of your work at your institution. 

a. Initiate discussions on program planning and policy 

b. Collaborate in program development 

c. Consult on impending policy changes 

d. Serve on planning and policy committees 

e. Present your work at executive level meetings 

f. Conduct follow-up studies on the impact of work 

g. Work is disseminated at the VP and Presidential Level 

h. Work is used in executive decision-making 

i. Work effects program and policy changes 

j. Work includes policy recommendations 

Job Quality (A=. 86) *** 

To what extent are the following items a rewarding part of your job? 

a. Freedom to decide how to do your work 

b. Being able to make decisions on your own 

c. Authority you need to get the job done 

d. Being able to work on your own 

e. Authority to set your own research agenda 

f. Flexibility to establish your work priorities 

g. Freedom to decide how your work will be shared 

h. Freedom to accept or reject superior’s suggestions 

i. Independent authority to hire persons of your choice 

j. Authority to spend department budget as you wish 

k. Supervisory support for professional development 

l. Financial support for professional development 



*** Response Scale: 1 Almost Never' to 5 ’Very Frequently' 
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The impetus for this research was an urgent need for comparative data on the typical 
staffing and responsibilities of institutional research (IR) offices. The author wished to 
obtain data that would enable officials at his university to judge whether the size of the 
staff of the IR office was commensurate with its objectives. To make such an assessment, 
it was necessary to know the responsibilities that would accompany given staffing levels 
at other colleges. Furthermore, because Fordham has both a Catholic and Jesuit identity, 
it was desirable to obtain data from schools with similar traditions. 

Several regional studies have examined the size and responsibilities of IR offices. 
They have generally found that the enrollment of a school influences the size of the IR 
department, which in turn affects the complexity and sophistication of the analytical tasks 
that the IR office conducts (Delaney, 1997; Volkwein, 1990). There appears to be a 
common core of activities that most institutional research offices perform (Muffo, 1999; 
Volkwein). 

One must be cautious about generalizing from these studies. None reported separate 
statistics for Catholic or Jesuit institutions. Those with samples drawn from institutional 
research associations may not be representative of colleges that do not belong to such 
organizations. In a comparison of North American regional studies, Muffo (1999) 
observed that regions differ in the type, control, and size of schools, as well as the 
requirements of accrediting organizations. He noted too that the dominance of 
enrollment research in IR offices in the northeast and New England might reflect the 
efforts of colleges to cope with slow enrollment growth in these regions. 

This exploratory study has three purposes: (1) to obtain data on staffing and 
responsibilities for institutional research at Jesuit colleges and at other Catholic 
universities that are large or have significant doctoral programs, (2) to present data that 
would enable administrators at peer institutions to assess the adequacy of staffing for 
institutional research functions, and (3) to explore a methodology for determining the 
staff necessary to accomplish typical IR tasks. 

Method 

Participants. The target population for this survey included all Jesuit colleges and 
universities, as well as Catholic institutions that were large or that had significant 
doctoral programs. The investigator obtained responses from 23 of the 28 Jesuit colleges 
and universities in the U.S., from 10 of the 12 largest Catholic schools (DePaul 
University Enrollment Management Research, 1998), and from 10 of the 1 1 Catholic 





universities that participated in the 1995 rating of graduate programs by the National 
Research Council (Webster & Skinner, 1996). These categories overlap. Of 36 
institutions in the sampling frame, 31 (or 86 percent) participated in the study. 

Procedure. The investigator conducted a telephone survey of officials responsible for 
institutional research during the summer and early fall of 2000. He promised not to 
report information that might be identified with a respondent's school. 

Directors or coordinators of institutional research provided data for the 1999-2000 
academic year. They reported total headcount enrollment of the institution in fall 1999. 
They provided also headcount and full-time equivalent statistics for six categories of 
personnel: full- and part-time professionals, full- and part-time support and clerical 
workers, graduate assistants, and other student employees. Then, the respondents 
indicated whether their offices had performed each task on a checklist of IR projects. 

The investigator also gave participants an opportunity to identify responsibilities that 
were not included in the checklist. After conducting initial interviews, the researcher 
expanded the list. He obtained information from 31 schools on the activities in the initial 
questionnaire and from 19 colleges on the added items in the second phase of the survey. 

Some of the schools examined did not have institutional research offices. In such 
cases, the investigator obtained information from the administrator or faculty member 
who had the most responsibility for institutional research functions. 

The following report provides information on 29 schools. The investigator did not 
include the two largest institutions that responded to the survey because readers probably 
would be able to identify the schools. 

Table 1 displays the number of schools in the sample according to headcount 
enrollment and total full-time-equivalent personnel. The number of participants in the 
second phase of the survey is given only by enrollment because no results for the second 
stage are reported by size of staff. 



Results 

The mean headcount enrollment of the institutions participating in the survey was 
7,357 (SD = 3,243). The average size of IR offices was 2.9 full-time-equivalent (FTE) 
persons (SD = 1.8). The data on personnel were combined into three categories: full-time 
professionals, other employees (part-time professionals and full- and part-time clerical 
and support staff), and student workers (graduate assistants and other students). 

Figure 1 shows that the average size of a staff increases with enrollment and that 
student workers make up only a small proportion of the FTE staff. A few IR directors 
commented that it was not efficient to use students because they are temporary and part- 
time and require extensive training. 



Table 1 

Number of Schools in Sample by IR Staff Size and Enrollment 

Headcount Enrollment 



Sample and 



FTE Staff 


< 5,000 


5,000-9,999 


10,000-14,999 


Total 


Full 


<3 


5 


9 


4 


18 


3 to 5.99 


2 


4 


2 


8 


> = 6 




1 


2 


3 


Total 


7 


14 


8 


29 


Phase 2 


Total 


6 


9 


4 


19 
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Figure 1 . Full-Time-Equivalent IR Staff at Catholic Colleges and 
Universities by Enrollment (Headcount), 1 999-2000 (N = 29). 



An alpha level of .05 was used for all statistical tests. 

To gain greater insight into the relation of office size to enrollment and institutional 
complexity, the investigator completed a regression of FTE staff against headcount 
enrollment and the Carnegie classification as revised in 1994 ("Carnegie Foundation's 
classification," 1994). To correct for heteroscedasticity, a generalized least squares 
(weighted) regression model was used. There was no significant relationship between 
Carnegie classification and FTE personnel. The full and reduced models appear in table 
2. See figure 2 for a plot of predicted FTE staff against headcount enrollment as 
developed in the reduced model. 



Table 2 

Summary of Simultaneous Weighted Regression Analysis for Variables Predicting 

FTE Staff in Institutional Research Functions (N = 29) 

Full Model Reduced Model 



Constant 

Headcount Enrollment 
Carnegie Classification 
Adj. R 2 

SE 

F 



B SEB 

1.09951 0.65321 

0.00045 * 0.00013 
-0.30955 0.20916 

0.29 

0.86 

6.74* 



B SEB 

0.52187 0.53521 

0.00033* 0.00010 

0.26 

0.88 

10.82* 



Note. * p < .05. 




Headcount Enrollment 

Figure 2. Predicted FTE Staff in IR Offices (1999-2000) 



To facilitate a comparison with Volkwein’s (1990) results, FTE staff size was 
correlated with (1) headcount enrollment and (2) Carnegie classification. Significant 
correlations were obtained with headcount enrollment, r(29) = .46 and Carnegie 
classification, r(29) = .45. Volkwein obtained .73 and .60, respectively. Using Fisher's Z 
transformation, one can test the difference between correlations from two independent 
samples. The difference between the (a) the correlation between headcount and FTE staff 
obtained here and (b) that of Volkwein is significant, x 2 (l) = 68.56. The beta weight for 
enrollment regressed against FTE staff in the reduced regression model in this study was 
.54. 



What tasks do IR offices perform? Table 3 shows the percentage of schools that 
report engaging in an activity by size of school. For tasks listed in the full sample, the 
table provides also percentages by full-time-equivalent staff size. Thirteen of the tasks 
listed by Volkwein (1990) had labels similar to the titles used in this study. The last 




64 



56 



column shows that percentage of institutions that reported completing similarly described 
tasks in Volkwein’s report (1990). 



Table 3 

Percentage of IR Offices Completing Activities by Size of Institution and IR Staff 



Activity and FTE Staff 


Headcount Enrollment 
5,000- 10,000- 
<5000 9,999 14,999 Total 


Volk- 

wein a 


Responding to Surveys 












IPEDS Reports (F) 












< 3 FTE's 


80% 


100% 


75% 


89% 




3 - 5.9 FTE's 


100% 


75% 


100% 


88% 




> = 6 FTE's 




100% 


100% 


100% 




Total 


86% 


93% 


88% 


90% 


85% 


Major Surveys, e.g., US News, NCAA (F) 












< 3 FTE's 


100% 


100% 


75% 


94% 




3 - 5.9 FTE's 


100% 


75% 


100% 


88% 




> = 6 FTE's 




100% 


100% 


100% 




Total 


100% 


93% 


88% 


93% 




Minor Surveys, e.g., College Guides (F) 












< 3 FTE’s 


60% 


89% 


25% 


67% 




3 - 5.9 FTE's 


100% 


75% 


100% 


88% 




> = 6 FTE's 




100% 


100% 


100% 




Total 


71% 


86% 


63% 


76% 


81% 


Total Major and Minor Surveys (F) 












< 3 FTE’s 


100% 


100% 


75% 


94% 




3 - 5.9 FTE’s 


100% 


75% 


100% 


88% 




> = 6 FTE's 




100% 


100% 


100% 




Total 


100% 


93% 


88% 


93% 




Fact Book (F) 












<3 FTE’s 


60% 


89% 


50% 


72% 




3 - 5.9 FTE’s 


50% 


50% 


100% 


62% 




> = 6 FTE’s 




100% 


100% 


67% 




Total 


57% 


71% 


75% 


69% 


77% 


Retention Analysis (F) 












<3 FTE’s 


100% 


89% 


100% 


94% 




3 - 5.9 FTE’s 


50% 


100% 


100% 


88% 




> = 6 FTE’s 




0% 


100% 


67% 




Total 


86% 


86% 


100% 


90% 


93% 
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Table 3 (continued) 





Headcount Enrollment 








5,000- 


10,000- 




Volk- 


Activity and FTE Staff 


<5000 


9,999 


14,999 


Total 


wein a 


Financial Aid and Tuition Discount Analysis 












(F) 












< 3 FTE's 


60% 


44% 


100% 


61% 




3 - 5.9 FTE's 


100% 


75% 


50% 


75% 




> = 6 FTE's 




100% 


100% 


100% 




Total 


71% 


57% 


88% 


69% 




Enrollment Management 












Admissions: Performance Monitoring (S) 


100% 


33% 


75% 


63% 




Admissions: Operational Support (S) 
Admissions: Marketing Research or 


0% 


11% 


50% 


16% 




Policy Analysis (S) 

Admissions: Research or Support (F) 


17% 


56% 


75% 


47% 




< 3 FTE's 


0% 


22% 


75% 


28% 




3 - 5.9 FTE's 


50% 


75% 


50% 


62% 




> = 6 FTE's 




100% 


50% 


67% 




Total 


14% 


43% 


63% 


41% 




Enrollment Projections (S) 

Information System Policy Development (F) 


67% 


78% 


100% 


79% 


80% 


< 3 FTE's 


60% 


100% 


75% 


83% 




3 - 5.9 FTE's 


50% 


75% 


100% 


75% 




> = 6 FTE's 




100% 


50% 


67% 




Total 


57% 


93% 


75% 


79% 




Assessment Surveys— Satisfaction; Cognitive, 












Personal, Career Development; Alumni 

(F) 












< 3 FTE's 


60% 


78% 


50% 


67% 




3 - 5.9 FTE's 


100% 


100% 


100% 


100% 




> = 6 FTE's 




100% 


100% 


100% 




Total 


71% 


86% 


75% 


79% 




Academic Program Review (S) 

Participate in Regional Accreditation Self- 


83% 


22% 


50% 


47% 


33% 


Studies (F) 












< 3 FTE's 


80% 


100% 


75% 


89% 




3 - 5.9 FTE's 


100% 


75% 


100% 


88% 




> = 6 FTE's 




100% 


100% 


100% 




Total 


86% 


93% 


88% 


90% * 


48% 





Table 3 (continued) 





Headcount Enrollment 








5,000 - 


10,000 - 




Volk- 


Activity and FTE Staff 


<5000 


9,999 


14,999 


Total 


wein 3 


Faculty Analyses 












Faculty Load Analysis (F) 












< 3 FTE's 


80% 


78% 


25% 


67% 




3 - 5.9 FTE's 


50% 


50% 


100% 


62% 




> = 6 FTE's 




100% 


100% 


100% 




Total 


71% 


71% 


62% 


69% 


76% 


Faculty Flow Analysis (S) 


33% 


11% 


0% 


16% 




Faculty Compensation (S) 


67% 


56% 


25% 


53% 


64% 


Participation in Strategic Planning (F) 












< 3 FTE's 


80% 


89% 


75% 


83% 




3-5.9 FTE's 


100% 


75% 


100% 


88% 




> = 6 FTE's 




100% 


50% 


67% 




Total 


86% 


86% 


75% 


83% 




Peer Analyses — Benchmarking Studies (S) 
Environmental Scanning (F) 


100% 


67% 


50% 


74% 




<3 FTE's 


40% 


22% 


0% 


22% 




3 - 5.9 FTE's 


50% 


0% 


50% 


25% 




> = 6 FTE's 




100% 


50% 


67% 




Total 


43% 


21% 


25% 


28%* 


67% 


Cost Analyses (S) 


33% 


44% 


50% 


42% 


44% 



Note. F = full sample (N= 29); S = second phase or small subsample (N = 19). 



“Volkwein (1990). 

*Difference between totals in this study and Volkwein (1990) is significant, p < .05. 

The percentages in the total column of table 3 indicate that more than 90 percent of 
IR offices participate in regional accreditation self-studies, complete IPEDS reports, 
respond to surveys, and conduct retention analyses. A minority of IR offices provide 
operational support or marketing research for admissions programs, participate in 
academic program review, analyze faculty flow, do environmental scanning, or perform 
cost analyses. 

In general, the proportions of Jesuit and Catholic colleges engaging in activities were 
not different from those listed in Volkwein’s (1990) article. However, two-tailed tests of 
differences in proportions were significant for two activities. Namely, the schools in this 
survey were less likely than those in Volkwein’s study to engage in environmental 
scanning, but more likely to participate in regional accreditation studies. 

Because of small cell sizes, the investigator did not conduct statistical tests of 
association between institutional size, staff size, and performance of IR tasks. 
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Nevertheless, several activities appear to be related to size of college. The DR offices at 
larger institutions are more likely to provide operational or research support to 
admissions offices, to complete enrollment projections, and to conduct cost analyses. 
Institutional researchers at small colleges are more likely to complete benchmarking 
studies and faculty analyses than their counterparts at large schools. 

The data suggest several activities that are related to staff size. Small DR offices tend 
not to complete minor surveys for college guides and the like. This is particularly true at 
large schools. Rather, the offices of public affairs and admissions have responsibilities 
for such surveys. DR offices with small staffs are less likely than large offices to perform 
financial aid and tuition discount analyses, to conduct assessment surveys, and to do 
environmental scanning. The completion of fact books appears to be related to both 
school and IR staff size. The proportion of offices compiling fact books increases with 
size of college and size of DR staff. Furthermore, small DR offices in large colleges are 
among the least likely DR departments to complete fact books. 

Respondents to the survey identified general responsibilities that were not included in 
the structured questionnaire. Table 4 lists the items mentioned. Except for policy and 
management studies, very few institutions performed any one of the activities. 
Furthermore, most of the projects were conducted in the largest IR offices in the survey. 

Discussion 

The average enrollment of institutions that participated in this investigation is in the 
middle range of the regional surveys summarized by Muffo (1999), as is the average size 
of institutional research staffs. In this study, the correlation between staff size and 
enrollment is much smaller than what Volkwein (1990) obtained (.46 vs. .73). However, 
the correlation obtained in the reduced (bivariate) regression model in this study was .54. 
These results suggest that enrollment has a moderate influence on the size of IR staffs in 
Catholic and Jesuit colleges, but to a lesser extent than is typical of institutions in the 
northeast. This report has used Carnegie classification as a measure of the breadth of 
degree programs and institutional complexity. The correlation between this measure and 
FTE staff and what Volkwein found (.45 vs. .60) was not significant. It would appear 
that among Catholic and Jesuit institutions, factors apart from enrollment, extent of 
degree programs, and institutional complexity determine the size of IR offices. 

Most of the institutions in this survey completed a core set of tasks, which included 
IPEDS reports, surveys from outside organizations, and retention analyses. With three 
exceptions, the DR tasks performed present a profile like that of institutions in North East 
Association for Institutional Research (NEADR; Volkwein, 1990). Unlike NEADR 
members, the DR officers at Catholic and Jesuit schools were less likely to do 
environmental scanning and more likely to participate in regional accreditation studies. 

Some data obtained in this survey suggest that the size of a college might be related to 
DR responsibilities. A greater percentage of large IR offices at large universities compiled 
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fact books than IR departments at small colleges. It may be that large institutions require 
more formal means of disseminating information than small schools. 



Table 4 

Responsibilities Not Included in Structured Questions 



Policy and management studies (ad hoc studies) 

Data and Information Management 

Cleaning and auditing data for all university academic systems 
Data warehouse development and operation 

Develop programs to extract data from the university information system 
Implementation of new university-wide information systems 
Academic Management 
Faculty contracts 
Course flow 

Classroom utilization management 

Analysis of productivity of departments and schools 

Editing of catalog 

Academic program and faculty development 
Organizational Development 

Departmental consultation and team building 
Coordinate Quality Improvement Program 

Coordinate all training and professional development programs of the university 
Facilitate discussions on academic issues (surveys and focus groups) 

Develop an analytic culture in the university 
Information Collection and Dissemination 
Community service report 
Faculty publications list 
Additional Assessment Activities 

Teacher and course evaluations 
Staff satisfaction surveys 
Student, faculty, and staff climate surveys 
Survey Design Consultation 
Financial and Management Analysis 
Tuition and fee policies 
Forecasting revenue from tuition and fees 
Budget models 

Work force analysis 



Some activities are uncharacteristic of IR offices with small staffs. These include 
compiling fact books, completing small surveys, analyzing financial aid and tuition 
discount programs, conducting assessment surveys, and doing environmental scanning. 
Many of these activities are either labor intensive or highly sophisticated. Perhaps core 
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IR activities take up most of the time of a small staff. A relatively large IR staff may be 
necessary to conduct labor intensive or highly specialized work. 

After controlling for enrollment, colleges in this study show considerable variation in 
IR staff size. Nevertheless, the models in this report enable IR officers at Jesuit and 
Catholic colleges to calculate an expected staff size based on enrollment. Furthermore, 
the data in tables 3 and 4 can be used to determine if a school is different from its peers 
because the ER office devotes resources to atypical activities or because it fails to 
complete common tasks. 

What are the limitations of this study and what are implications for future research? 
The sample size in this survey is small. However, it is important to recall that the 
respondents constituted 86 percent of the target population. Any extension of this work 
to a fuller range of colleges and universities would enable an investigator to increase 
sample size. In addition, any subsequent study should consider indicators of complexity 
other than Carnegie classification. These might include the number of schools and 
campuses within a university. It would be beneficial to emulate Volkwein’s (1990) 
analysis by obtaining data (1) on the organizational location of IR offices and (2) on the 
extent to which tasks are shared with other offices. Finally, to develop standards for what 
is required to complete IR tasks, it might be best to focus on the activity as the unit of 
analysis and to collect information on the skill level and amount of time required to 
complete major IR projects (Personal communication, Thomas J. Dimieri, November 6, 
2000 ). 
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Introduction 

Higher education institutions are facing a technological revolution in almost every 
aspect of operation. Universities are rushing to increase offerings of on-line courses, on- 
line course registration, automated advising systems, and to provide infrastructures to 
accommodate the growing computer and telecommunication needs of its faculty, staff 
and students. What makes managing this revolution even more difficult is the fact that 
the people leading the revolution on the user-end are high school and college-aged 
students, who bring their sophisticated technological habits to campus. Frand (2000) 
notes that in 1998, for the first time since television was introduced to the public, the 
number of hours young people spent watching television decreased. This decrease was 
due to the increased time spent on the Internet. The Web is now the prime information 
source for young people. However, unlike television, the Internet is an interactive 
medium. Young people are now communicating more than ever, whether it be through e- 
mail, instant communication, or bulletin boards. In fact, it has recently been reported that 
the average connected American sends at least one email a day, spends on average 8.8 
hours per week online, and visits an average of 9 sites (Milliron & Miles, 2000). One 
would expect that these averages would be exceed for high school and college-age 
students 

In 1996, it was estimated that approximately 90 percent of college and university 
students in North America have ready Internet access, compared to less than one-tenth of 
the general population (Chidley, 1996). Four years later the landscape is dramatically 
different. According to a October 2000 report by the National Telecommunications and 
Information Administration, 51 percent of all US households had computers and of those 
households 80 percent have internet access. While a 2000 estimate for the percentage of 
college and university students with ready access to the Internet is not available, it is 
quite likely that it is has increased somewhat since 1996. Given that colleges and 
universities are now admitting students of the “NET generation”, it is imperative that 
institutions understand how prospective students as well as enrolled students interact with 
various members of the campus community. Current estimates are that 56.8 percent of 
individuals age 18-24 and 53.4 percent of youth age 9-17 use the internet (NTIA, 2000). 
One would expect that these numbers are likely to increase over the next five year. 
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Objective 



The primary objective of this paper is to examine how prospective students as well as 
current undergraduates are using electronic communication to interact with various 
campus constituencies at a Research I university. Data from three distinct surveys were 
used to understand this phenomenon: 1) Undergraduate Non-Enrolled Survey (accepted 
but did not enroll), 2) Undergraduate New Student Survey and 3) Graduating Senior Exit 
Survey. 



Literature 

There has been an abundance of research done on the effect electronic 
communication has on learning and socialization in college (Green, 1998; Duin & 
Archee, 1996; Ritter, 2000; Windschitl & Lesehm- Ackerman, 1997; Zagorsky, 1997). 
There is not, however, a body of literature on how students interact electronically with 
university constituents outside of the classroom. Research that informs this study are 
Selwyn’s (1998) study of 16-19 year olds’ domestic use of computers and the 
relationship with use of information technology in school or college, Fishman’s (1999) 
study on predicting students’ success with computer-mediated communication, and 
Piirto’s (1998) work on how college students use and view e-mail in regards to specific 
content communication. 



Data Sources and Methodology 



Accepted Applicant Surveys 

Undergraduate accepted applicants (non-enrolled students and new students) for the 
Fall semester were mailed surveys, with business-reply return envelopes, in May, 2000. 
Of the 1,183 entering first year students, 798 returned New Student surveys, resulting in a 
67.5 percent response rate. Of the 2197 students, who were accepted but did not enroll, 
795 returned Non-Enrolled Student surveys. Thus, the response rate for non- 
matriculating students is 36.2 percent. 

The admissions survey instruments, which have been administered over the past 
fifteen years, were augmented this year to include questions about electronic 
communications. The admissions office staff and dean were quite instrumental in 
providing “content” guidance. Survey items were in the form of categorical responses, 
Likert-type scales, and open-ended comments. The array of questions posed to incoming 
students and non-enrolled students were very comparable. With regard to the use of 
computers and the Internet, accepted applicants were queried to ascertain 1) software and 
communication applications used and extent of that usage, 2) Intemet/computer sources 
used to gather information about schools during the application process, 3) the 
characteristics of the computer hardware utilized to connect to the Internet, 4) reasons 
that prevented individuals from submitting their applications electronically, and 5) the 





importance of having a specific array of computing/electronic capabilities available to 
them at the institution. 

Senior Survey 

Data from the 2000 Senior Survey was used to determine the extent of e-mail and 
other electronic communication usage among currently enrolled students. Graduating 
seniors were asked to complete the survey during the week prior to graduation. 
Historically the response rate to this data gathering has been quite high since graduating 
senior must complete the survey in order to participate in commencement ceremonies. 

Of the 1,183 graduating seniors, 1,131 submitted completed surveys for a response rate of 
96 percent. 

In addition to the standard items that have been used for the past ten years, new items 
were developed with the purpose of gaining a better understanding of students’ electronic 
communication usage. Survey items were in the form of categorical responses, Likert- 
type scales, and open-ended comments. Graduating seniors were asked a variety of 
questions regarding with whom and how often they communicated via e-mail. In 
addition, they were asked the frequency with which they participated in academic 
listserves, electronic chat room and used the internet for research or classroom 
assignments. 



Analysis 

Data from the surveys were analyzed in SPSS by running frequencies on the relevant 
variable items. Open-ended comments related to relevant items were also organized and 
analyzed. Analysis of open-ended items consisted of coding and categorizing responses. 

Findings 



Accepted Applicants 

Generally, accepted applicants used the Web, e-mail, and instant communication - 
also known as interactive real-time chat (IRC) — extensively (see Table 1). Over 78 
percent of the respondents used e-mail at least once per day, and 70 percent accessed the 
Web at least once per day. Surprisingly, 59 percent of the respondents stated that they 
used IRC at least once per day. And, while the use of online shopping on a regular basis 
was minimal, over half (56.2%) had shopped online at some time. 



TABLE 1 



GENERAL USE OF 


ELECTRONIC COMMUNICATION AND THE 
INTERNET 




i 


Several times 
per day 


Once per 
day 


Once per 
week 


Once per 
month 


Few times 
per year 


Never 


1 . Web Access 


33.5% 


36.6% 


19.5% 


4.5% 


1 .7% 


4.2% 


2. E-Mail 


35.0% 


43.7% 


15.9% 


2.1% 


2.1% 


1 .2% 


3. Instant 
Communication 


32.8% 


26.1% 


14.4% 


5.2% 


5.4% 


16.2% 


4. On-Line 
Shopping 


0.1% 


0.3% 


5.0% 


16.7% 


34.1% 


43.8% 



Accepted applicants were asked to identify the methods they used to make initial 
contact with the institution. As students began collecting information in order to 
determine where they would apply, they most frequently accessed college web pages 
(58.4%). Visiting campus was the second most popular method of collecting initial 
information (33.9%), followed by email contact with admissions offices (24.3%) and 
calling admissions offices (23.8%) (see Table 2). 



TABLE 2 



FIRST CONTACT WITH COLLEGES TO 
COLLECT INFORMATION 




% 


1 . Accessed College Web Pages 


58.4% 


2. Called Admissions Offices 


23.8% 


3. Wrote to Admissions Offices 


1 1 .8% 


4. Emailed Admissions Offices 


24.3% 


5. Visited Campus 


33.9% 


6. Other 


8.2% 
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In addition, accepted applicants were asked to identify specific aspects of the Web or 
Internet that were used to help decide where to apply. The most frequently cited resource 
was individual college and university WWW pages, as stated by 82% of the respondents. 
Almost 29 percent of the students also stated that they relied on the U.S. News & World 
Report WWW site, and almost 28 percent used general college information sites (see 
Table 3). Only 14 percent of the accepted applicants indicated that they did not use web 
or Internet resources to aid in their application decisions. 



TABLE 3 



ASPECTS OF WEB OR INTERNET USED TO HELP DECIDE 


WHERE TO APPLY 


% 

82.0% 


1 . Specific College/University WWW Pages 


2. General College Information Sites 


27.9% 


3. US News & World Report WWW site 


28.7% 


4. Other WWW site 


4.5% 


5. Did not use the WWW or Internet in my 
application decision 


13.8% 



In regards to learning specifically about Tufts, over 77 percent of accepted applicants 
used the admissions web site. However, only 31 percent of the accepted applicants 
indicated that they had e-mail contact with the Tufts admissions office and just over 21 
percent took the Virtual Tour on the Tufts WWW page. Like many institutions, Tufts has 
a web-based application process, however, only 18.2 % of the accepted applicants 
utilized it to submit their applications (see Table 4). 



TABLE 4 



USE ELECTRONIC COMMUNICATION AND THE INTERNET 
TO COMMUNICATE WITH, AND TO LEARN ABOUT TUFTS 




YES 


NO 


1 . Accessed Admissions Web Site 


77.1% 


22.9% 


2. Email Contact with Admissions 


30.9% 


69.9% 


3. Took Virtual Tour 


21.1% 


78.9% 


4. Submitted an Electronic Application 


18.2% 


81.8% 



Those applicants who did not submit an application electronically, through the web, 
were asked why. Over two-thirds of the respondents (68.5%) stated that paper 
applications seemed more reliable, and over 36 percent felt that an electronic application 
was too impersonal (see Table 5). A significant number of students were also concerned 
that they did not know how the application would look when it arrived at the Admissions 
Office (23.1%). 



TABLE 5 

WHY AN ELECTRONIC APPLICATION WAS NOT SUBMITTED 



% 

1 . Did not have the computer/WWW skills 3.4% 

2. Was not aware that electronic submission was an 8.2% 

option 

3. Started applying electronically, but encountered 6.4% 

technical problems 

4. Did not have access to a personal computer 0.7% 

5. My computer couldn't handle the task 4.5% 

6. Too impersonal 36.1% 

7. Paper seemed more reliable 68.5% 

8. Didn't know how application would look when it arrived 23.1% 

at T ufts 

9. Thought I might be at a competitive disadvantage 12.4% 

10. Parents/friends preferred that I submit applications on 20.2% 
paper 

1 1 . My high school required me to submit an application on 7.4% 
paper 

12. Don't like using credit card via the WWW 16.2% 

1 3. Other concern 1 1 .0% 



Accepted applicants were also asked to identify the computer and Internet capabilities 
they felt were important aspects of a college campus environment. Intemet/WWW 
access and e-mail were, by a wide margin, considered to be the most important 
capabilities. Over 97 percent felt that Intemet/WWW access was either essential or very 
important, and over 94 percent felt that e-mail was either essential or very important. 
Relative to the other capabilities, students did not feel that online study/discussion groups 
were an important capability. Only 27 percent of the respondents indicated that this 
function was essential or very important (see Table 6). 



TABLE 6 

IMPORTANCE OF INSTITUTIONAL COMPUTER AND INTERNET 

CAPABILITIES 



1 . E-mail 

2. I nte met/WWW access 

3. Library access from the WWW 

4. Electronic class registration 

5. Online study/discussion groups 

6. Online course descriptions 

7. Online information access 
(grades, etc.) 



Essential 


Very Important 


Somewhat 

Important 


Not At All 
Important 


78.5% 


16.3% 


4.2% 


1 .0% 


85.1% 


12.0% 


2.8% 


0.2% 


44.7% 


37.7% 


15.7% 


1 .9% 


15.7% 


36.7% 


37.0% 


10.5% 


7.9% 


19.2% 


46.7% 


26.2% 


25.7% 


41 .6% 


26.7% 


5.9% 


26.7% 


40.1% 


26.4% 


6.0% 



Graduating Seniors 

As with the accepted applicants, graduating seniors used e-mail extensively on a 
regular basis during their senior year. Over 91 percent identified communicating by e- 
mail with other Tufts students at least once per week, and almost 80 percent 
communicated with students at other colleges at least once per week. Only 52 percent, 
however, communicated with faculty at least once per week. And while over 95 percent 
used the Internet for research or homework, only 44 percent had participated in an 
academic listserv. A significant number of graduating seniors (76.9%) also had not 
participated in an electronic chatroom (see Table 7). 
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TABLE 7 

USE OF ELECTRONIC COMMUNICATION BY STUDENTS DURING 

SENIOR YEAR 





Daily 


1 . Communicate with Tufts 
faculty 


5.3% 


2. Communicate with Tufts 
students 


61.3% 


3. Communicate with faculty at 
other colleges 


1.6% 


4. Communicate with students 
at 

other colleges 


36.4% 


5. Communicate with friends 


48.6% 


6. Communicate with family 


23.0% 


7. Participate in an academic 
listserv 


8.4% 


8. Use the Internet for 
research or 
homework 


21.6% 


9. Participate in electronic 
chatrooms 


4.4% 


10. Other Internet use 


52.3% 



2-3 times per 
week 


Once per 
week 


1-2 times per 
month 


Never 


18.8% 


28.0% 


45.1% 


2.8% 


21.1% 


9.0% 


5.7% 


2.9% 


2.1% 


3.6% 


19.3% 


73.4% 


27.1% 


16.4% 


14.8% 


5.3% 


25.2% 


13.9% 


9.7% 


2.7% 


25.7% 


20.0% 


18.5% 


12.7% 


9.8% 


10.8% 


14.6% 


56.5% 


31.1% 


20.2% 


22.9% 


4.1% 


3.1% 


4.2% 


1 1 .4% 


76.9% 


21.9% 


12.8% 


7.8% 


5.1% 



Conclusions 

Results of this study verify the extensive use of and reliance on electronic 
communication through the Internet by college-bound students and by students currently 
enrolled at Tufts. Both accepted applicants and graduating seniors use e-mail extensively 
to communicate with friends and other students, and a majority of college-bound students 
are now using instant communication on a regular basis. All groups are also using the 
Web extensively as an information source and to conduct research and complete 





homework assignments. Virtually all accepted applicants noted the great importance of a 
campus’ e-mail and Internet capabilities. 

Students are not, however, using the Internet extensively to communicate with Tufts. 
Less than one-third of applicants communicated by e-mail with the Admissions Office 
during the application process, and less than one-fifth of applicants submitted an 
electronic application. And while the submission of an electronic applications has added 
risks, such as the uncertainty of the technology in submitting a competitive application or 
the use of a credit card online, the fact that over half of the accepted applicants had 
shopped online shows that they have experienced the technology and have addressed 
those risks. Graduating seniors also noted much less e-mail use to communicate with 
faculty than they used to communicate with other students. 

Results also show that there is a lack of interest, among both groups, in web-based 
group activities. Accepted applicants did not rate online discussion groups as an 
important capability on campus, and graduating seniors did not participate in listservs or 
chat rooms to a large extent. 



Implications 

These findings demonstrate that currently there appears to be a gap between the 
general use of electronic communications among undergraduates and college-bound 
students and their use of this technology to communicate and interact with the institution. 
At this time there is relatively little information available to help faculty and 
administrators to understand why these gaps exist. Identification of such a divide may 
serve as an impetus to explore whether current policies, practices, or infrastructure are 
impediments to electronic communication. In addition, the findings suggest that 
institutions my want to examine the use of newer communication technologies - such as 
IRC - and how they may be utilized to increase the degree of contact with current and 
potential students. 

Currently the information that exists regarding computer choices and computing skills 
of college freshmen is very limited (Olsen, 2000). If institutions are to be effective in 
providing electronic forms of communication opportunities, the time has come when the 
higher education community must obtain a clearer understanding of 1 ) students level of 
computer use proficiency, 2) their preferences regarding electronic communication, 3) 
factors that prevent the use of electronic communication with specific populations, and 4) 
reasons that preclude a subset of the population from utilizing these new technologies. 
Future survey research endeavors at Tufts will include additional items designed to shed 
additional light on questions surrounding use and non-use of electronic communication. 
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Introduction 

The Admitted Student Questionnaire™ (ASQ) and the Admitted Student 
Questionnaire Plus™ (ASQ+) are college choice surveys sponsored by the College 
Board. The ASQ, first offered in 1988, asks students to compare the college that mailed 
them the survey with the set of other colleges they seriously considered attending. The 
ASQ+, begun in 1992, asks students to name and rate two specific colleges in addition to 
the one initiating the survey. More than 200 colleges participate in the service each year, 
of which more than 85% have participated more than once. 

Despite the rise of Web-based surveys, the paper ASQ has shown no dropoff in 
participation. Nevertheless, the handwriting appears to be on the monitor, and the 
College Board has begun to consider whether the paper ASQ should be replaced, or at 
least supplemented, by a Web-based version. This paper describes the Board’s first 
efforts to evaluate the feasibility and effectiveness of an on-line ASQ+. 

Development of the project 

The first discussions about a Web-based ASQ+ began on January 17, 2000 between 
Applied Educational Research (AER) and Logicat, Inc. (both of which are subcontractors 
for the College Board on this project). Things moved very quickly after that to plan a 
pilot study that could be conducted during the current ASQ/ASQ+ cycle. Logicat had the 
responsibility to create the actual Web-based survey and to work out the details of 
hosting it, collecting tracking and evaluation information, and converting the data into a 
format that would be compatible with the ASCII files that are currently used in the 
analysis of ASQ/ASQ+ data. 

AER’s role was to act as liaison with the colleges chosen as participants in the pilot 
study. Three participating colleges were recruited. All three institutions conducted an 
ASQ+ study in 1999, and would normally not have participated in 2000 (College A and 
College C tend to participate every other year, and College B every three years). The 
contacts were asked to participate in a regular (i.e., paper) ASQ+ survey, with the added 
wrinkle that their admitted students would be offered the opportunity and encouraged to 
complete the survey on the Web. All of the study costs that would normally be paid to 
the College Board (questionnaire printing and processing, participation fee, local question 
fee, data CD) would be borne by the Board, leaving the pilot colleges with only the actual 
mailing costs to cover. 
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The target date for the survey to go “live” (i.e., be available to students on-line) was 
June 1 . The programmers were able to write the basic survey in about three weeks, and 
the time between that initial version and June 1 was spent refining the instrument and 
data verification procedures. During that period the three pilot colleges were asked to 
provide lists of Social Security numbers and names of the students who would be 
authorized to enter the Web site. They were also directed to prepare email lists for the 
students they would survey. An email message would go out once the site was ready 
telling the students about the ASQ and providing a direct link to each college’s survey 
site. 



Study Design Issues 

In an interest survey conducted by email in early 1 999, the greatest concerns about a 
Web ASQ expressed by recent ASQ participants were whether they would be able to get 
as much information from a Web version as they currently get from the paper survey, and 
whether response rates would suffer with a Web survey. While there were many 
variables that could be studied in the pilot — access to the Web and to computers, 
response rates, completion time, necessity for and effect of incentives, participating 
colleges’ access to their ASQ data, ASQ vs. ASQ+, products (reports), etc. — in order 
both to keep this pilot manageable given the short timeframe for development and to 
minimize ambiguity of results, this pilot was to have very limited objectives. In 
particular: 

• The reports would be identical to those the colleges currently receive. 

• The pilot would use the ASQ+ only (and not the ASQ). 

• The survey itself would mimic the current ASQ+ as closely as possible, given 
the use of survey techniques appropriate to the Web. 

• Pilot colleges would be drawn only from the pool of past ASQ+ users, in 
order to have baseline response rates for comparison. 

• The Web survey would not permit colleges to ask “local questions” (extra 
questions devised by the colleges) at this time. 

Questionnaire Design Issues 

In general, it proved quite easy to translate the ASQ+ survey to the Web. Each page 
of the paper survey became a separate Web page; clicking on a “next” button brought up 
another page. Students were able to go back and forward between pages, and could leave 
the survey altogether and return later until the point when they clicked on the “submit” 
button. The final page of the regular survey was followed by a few questions evaluating 
the survey process. The “submit” button was located at the bottom of this last page. 

The section of the survey that presented the greatest programming difficulties was 
that dealing with colleges to which the student applied. The principal concern, of course, 
was to design a survey form that would encourage the students to fill it out completely. 
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What then was the best way to collect information on colleges applied to by the 
responding students? The paper survey asks students to write in the name of the colleges 
to which they applied, along with the schools’ city and state (to aid in identification of the 
colleges at the data entry end). One alternative for the Web survey was to have the 
students do essentially the same thing: type in the name and location of the schools to 
which they applied. The major alternative to that method was to provide a drop-down list 
of institutions from which the student could select the ones he/she wanted to include in 
the survey. The latter method had the advantages that the students would have less 
information to enter themselves, and that there would be a significantly smaller chance of 
data entry errors. 

On the other hand, there was the question of how much patience students would have 
to search through up to 12 drop-down lists, i.e., whether they would give up at some point 
in that process and fail to complete the rest of the questionnaire. Even if the students 
were asked to enter each college’s state, so as to limit the number of schools included in 
the drop-down lists to those in the state specified, they could still be presented with a list 
of over 200 schools if they entered “California,” for example. It could, in fact, require 
more time for the student to complete this portion of the survey using drop-down lists 
than using a more old-fashioned write-in method. The author felt very strongly that the 
write-in method was preferable, and the programmers agreed to program the survey 
accordingly. 

Once the write-in method was agreed upon, the burden of verifying the accuracy of 
the students’ entries fell primarily on the programmers. Due to time and budget 
constraints, the initial program looked for an exact match between the student’s entry and 
the name of the college as stored in the College Board’s Annual Survey of Colleges 
(ASC) data file. Anything less than an exact match would produce an error on the 
college entry “validation page” that was included as part of the Web survey. (For any 
entry that did not produce an exact match, the students were presented with a drop-down 
list of institutions in the given state that began with the same letter as their entry.) Based 
on past experience with the paper survey, where students frequently write in nicknames 
such as “UVM,” “Sewanee”, or “Ole Miss,” the college list was expanded to include as 
many nicknames and variations on the official name as possible. 

While the matching difficulties seemed to have been resolved before the site went 
live, real time student entries demonstrated that the lookup process would still have to be 
revised. For example, if the University of Illinois at Urbana was entered by the student as 
the University of Illinois, Urbana, or the University of Illinois at Urbana-Champaign, or 
the University of Blinois-Urbana an error message would be generated, even though to 
the student’s eye these were clearly the same institution. The programmers decided to go 
back and modify the programming to ignore punctuation and such prepositions as “at”, 
“of’, “in”, etc. They accomplished this very quickly, thereby reducing the number of 
false errors in student entries, but the problem was not eliminated entirely. Ultimately, if 
the student could not or did not correct the entry on the validation page, the apparently 
erroneous entries were saved in a separate file to be dealt with at the time of analysis. 



The three pilot schools were given access to the test sites at the beginning of May. 
They all thought that the survey looked good, and had no suggestions to offer. 

Web Administration of the ASQ+ 

The target date for the Web site to go live was Thursday, June 1, 2000. The three 
pilot colleges were directed several times not to notify the students about the site until 
they received the go-ahead from the programmers, through AER. In addition, they were 
reminded several times that the paper survey should be mailed about a week after the 
email notification to the students, in the hope that this would maximize the number of 
students opting to respond via the Web. In the weeks prior to June 1 the schools were 
asked to provide lists of students who would be permitted to access the Web ASQ+. The 
information requested consisted of Social Security Number, to be used as the user ID, and 
the student’s first name, to be used as the password. A few students appeared to use their 
middle name more frequently than their first name, and the access information was 
adjusted accordingly. 

Prior to the grand opening Logicat also developed a “report” site, which would 
provide information on the site activity overall and for each school. There was also a 
page summarizing the students’ evaluations of the Web ASQ+. 

The first major problem surfaced about a month before the target date: College A 
discovered that applicant email addresses were not stored in a central electronic file, but 
merely in each student’s individual file. Thus there was no way to send out a mass email 
notifying students about the Web survey: College A would only be able to do the paper 
mailing. Since it was too late to recruit another school for the pilot, and since College A 
had already ordered and received its paper surveys, there was nothing for it but to 
continue as best we could. The college contact was instructed to make sure that the 
enclosure notifying the students about the Web option was printed on a separate, eye- 
catching sheet of paper. On the bright side, this study would provide some indication of 
the students’ willingness to take the extra step to go on-line specifically for the purpose 
of filling out the ASQ+, rather than simply clicking on the direct link to the survey that 
was to have been part of the email notification. 

Over Memorial Day weekend the programmers worked on transferring the survey to 
the external server they had chosen. During this period College A’s study produced 
another problem: the contact had decided that since College A would be doing a mail 
survey only, the mailing should take place before the site went live, so that students 
would be notified about the site in time for them to access it as soon as it was available. 
The introductory letter was dated May 24, and stated that the site would be available ”at 
the end of May.” In fact, some students apparently attempted to access the site as soon as 
they received the letter. The programmers discovered that questionnaires were being 
submitted while the site was still being tested and hosted at Logicat; those entries would 
have to be transferred to the “real” host later on. 
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On Friday, June 2, notification was received that the site was up and running for the 
students, and the contacts at the other two colleges were immediately sent the go-ahead 
for their mass student email. Activity was heavy for College C and College A over that 
first weekend, but very light for College B. Some students who experienced problems 
emailed the contacts, who forwarded the messages to AER. The messages were then 
passed on to the programmer, who responded very quickly. Some of the initial problems 
occurred in the section relating to colleges applied to, resulting in some of the 
programming changes described above. In a few cases student ID’s had not been 
included on the list of authorized respondents, but they were added as soon as the 
omission was discovered. 

At the beginning of the following week, activity was still almost non-existent on the 
College B site, raising concerns that there was a problem. The contact at College B also 
began to forward messages from students saying that their user ID (i.e., their SSN) and/or 
first name was not being accepted. On June 8 the contact sent an explanatory email: in a 
nutshell, the problem was that the lists of accepted and withdrawn students that were 
originally sent to the programmers did not match the lists of students who were sent the 
email notification about the Web option. College B produced new lists, and activity on 
that site immediately picked up. It should also be mentioned that the contact was 
concerned that non-enrolling students in particular were deleting messages from College 
B without reading them, and asked that the followup email come from AER. 

The final major problem encountered affected College C more than the other two. 
There was a bug in the survey that only manifested itself once the survey was moved 
from Logicat to the ISP server. Since a number of students logged on earlier than the 
programmers had anticipated, there wasn’t adequate time to test the survey on the ISP’s 
server. For more than 100 College C students, data from the sections on importance and 
quality ratings appeared to have been lost. (These students had gone all the way through 
the survey and clicked on the submit button, so it seems likely that they did finish, rather 
than submitting a survey with a lot of missing data.) This also occurred for about 40 
College A students, but College B’s respondents were unaffected because the problem 
was resolved by the time they were able to access the site. 

The best solution that could be devised was to reopen the surveys of the students 
affected (remember that they had locked their data originally by “submitting” their 
responses). Logicat compiled a list of the students to be contacted. College C provided 
their email addresses, and they were all sent a message explaining the project and the 
problem and asking for their helping in entering the missing data again. Although it was 
difficult to tell exactly how many students responded to this appeal, 40-60 appeared to 
have done so. 



Response Rates 

The results of the three ASQ+ studies have not been tabulated, but it appears that for 
College A and College B the response rates for the combined Web and paper versions of 
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the survey are close to but slightly lower than they were for the paper survey alone in 
1999. College C shows a higher response rate. Table 1 compares response rates at these 
three schools for all ASQ+ studies done since 1992. Note that the wide fluctuation in 
enrolling response rates at College B is due in part to timing of the survey’s 
administration: in the years when this rate was over 85%, College B administered the 
enrolling surveys to a captive audience at Freshman Orientation. Mailed surveys 
produced a much lower response. 



Table 1: ASQ+ Response Rate History 





College A 


College B 


College C 






Non- 






Non- 






Non- 






Enrolling 


Enrolling 


Total 


Enrolling 


Enrolling 


Total 


Enrolling 


Enrolling 


Total 


1991 


82% 


56% 


66% 


57% 


32% 


45% 








1992 


70% 


44% 


55% 


88% 


37% 


62% 








1993 














83% 


31% 


49% 


1994 








98% 


41% 


70% 








1995 


89% 


56% 


66% 








69% 


29% 


46% 


1996 








95% 


24% 


60% 








1997 


79% 


48% 


59% 








64% 


38% 


54% 


1999 


76% 


39% 


51% 


49% 


24% 


37% 


56% 


26% 


41% 


2000 

(total) 






50% 


43% 


25% 


35% 






49% 


2000 

(paper) 


57% 


33% 


41% 


31% 


20% 


26% 


50% 


25% 


37% 


2000 

(Web) 






9% 


11% 


5% 


9% 






12% 



N.B. None of these colleges participated in 1998. Enrolling/non-enrolling breakdowns are not yet 
available for Colleges A and C. 



Evaluation 

The contacts at the three participating colleges were asked to evaluate their 
experiences with the Web ASQ+ using a questionnaire emailed to them at the beginning 
of September. See Table 2 below for a summary of responses. 

It would be difficult to draw any conclusions on the basis of the these three studies 
only, since no college carried out the agreed-upon plan: email notification to students 
approximately one week before the paper mailing went out (including notification of the 
option to complete the survey on the Web), with email and paper follow-ups sent 
subsequently. Despite the shakedown problems at the beginning, all three contacts seem 
to have had a positive experience with the Web ASQ+. Once the final response rate 
information is available, however, they will be asked again. College A, in particular, has 
been quite concerned about the decline in non-enrolling response rates, and was hopeful 
that a Web option would alleviate that problem. 
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Table 3 summarizes student evaluations of the Web ASQ+. In general, the survey did 
not require much time at all, and was not perceived as too long or difficult to complete. 
Most of the students would fill out another survey like this one on the Web, and strongly 
prefer Web surveys to paper surveys in general. It is interesting that 28% of the people 
who received paper surveys said they did not fill any of them out. That figure, if 
accurate, represents an important target group for a Web-based ASQ+, especially if those 
are non-enrolling students. 

From AER’s point of view, the project went surprisingly easily for those students 
who chose the Web option, but the response rates were very disappointing. At this point, 
however, it would be very difficult to attribute the low response to any one thing: there 
was some type of problem - and a different problem — with each of the three studies that 
might have discouraged the students. Table 1 shows that ASQ response rates, especially 
from non-matriculants, have been declining steadily. The combination of paper and Web 
surveys may have helped slow this decline, although we do not have enough information 
to document this impression. 



Recommendations 

On the basis of both college and student comments it is clear that a Web ASQ is 
desirable. It is not clear, however, that the Web ASQ should completely replace the paper 
version. For one thing, good, working email addresses are still not universally available, 
at either the student or the college level. For another, questions about the make-up of the 
Web respondents have yet to be answered: are they representative of the total group of 
ASQ respondents in terms of both demographics and attitudes? Third, the issue of how to 
price a Web-based study has yet to be addressed. 

Nevertheless, the satisfaction levels of this year’s pilot colleges and their students 
were high enough to encourage continued development and testing. A second, expanded 
pilot would provide much more information on what to expect: the types of problems we 
could encounter on a regular basis, response rates from different types of institutions 
conducting the study somewhat differently (e.g., with the initial mailing/notification in 
July or August, rather than May or June), ease of dealing with the data at both the 
collection and analysis ends, etc. 
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Table 2: Participant Evaluations of the Web ASQ+ 
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Table 3: Student Evaluations of the Web ASQ+ 



1. Approximately how much time did you spend in completing this questionnaire? 

16.5 minutes 

2. Would you say the amount of time spent is: 



Acceptable 


69% 


Somewhat too long 


27% 


Much too long 


4% 



3. How would you rate the ease of entering your responses and moving through the 
questionnaire? 



Very Easy 


40% 


Fairly Easy 


32% 


About Right 


20% 


Fairly Cumbersome 


7% 


Very Cumbersome 


1% 



4. Would you complete another questionnaire presented on the web like this one if you were 
asked to by another college that had offered you admission? 



Yes, definitely 


23% 


Yes, probably 


56% 


No, probably not 


18% 


No, definitely not 


2% 


5. In the future, would you prefer to respond to this kind of questionnaire: 


In electronic form, like this one 82% 

In paper form, like a typical questionnaire in the mail 2% 


No preference 


15% 


6. Did you receive questionnaire(s) 


in the mail from any other college(s)? 


Yes 


42% 


No 


57% 


If Yes: Did you respond to: 




All of them 


35% 


Some of them 


37% 


None of them 


28% 


Total visits to Evaluation Page 


882 


Total surveys completed 


772 
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Table 4: Survey Participation, by College 
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CREATION OF A SCALE TO MEASURE FACULTY DEVELOPMENT NEEDS 
AND MOTIVATION TO PARTICIPATE IN DEVELOPMENT PROGRAMS 

Arthur Kramer 

Director, Institutional Research 
New Jersey City University 



Summary 

A survey was administered to full-time faculty to assess their perceptions of the 
University’s professional development program. Frequencies of responses were reported 
and the questionnaire responses were put through a factor analysis to explore the 
underlying qualities guiding the responses. 

The survey results showed tenured faculty desire experts in the disciplines be brought 
to campus to present findings of either the latest research in the discipline, or findings on 
the best/newest ways to teach the discipline. Twelve factors emerged of which six were 
judged to be interpretable. Communication surfaced as a large factor guiding the 
responses. Although the factor was judged positively, there were some specific areas in 
which greater communication was desired. These areas include communication of policy 
changes, on and off-campus opportunities, and planning of activities. The other factors 
concerned teaching and assessment, meetings, short and long term funding, and planning 
and usefulness of previous activities. 

Recommendations for future research that include assessment of components of 
personality stimulating participation in professional development activities and 
assessment of funding of such activities were brought out. 

Introduction 

The area of faculty development in higher education has grown from the initial 
implementation of sabbatical leaves, at Harvard in 1810, to structured programs targeting 
individual growth and vitality. Often, the differences among the initiatives are based on 
the missions of the institutions implementing them: teaching institutions often emphasize 
keeping current in the discipline and instructional strategies, while research institutions 
are more concerned with the performance of state-of-the-art research (Clark and 
Corcoran, 1989). Clark and Corcoran (1989) also note that faculty members at different 
stages in their career require, and often anticipate, different types of programs. This is 
because the career development needs and expectations of new faculty embarking on a 
career are different from the needs of tenured faculty in mid-career, or planning for 
retirement. This is in concordance with the beliefs of psychologists who theorize about 
different stages of peoples (e.g., Erikson, 1950; Super, 1984). For this reason, Clark and 
Corcoran (1989) advocate for programs addressing not only effectiveness in the 
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classroom and research, but also those incorporating life transitions, such as career 
counseling and retirement planning. 

An assessment of the strictly professional career development activities utilized by 
colleges and universities found that most brought guest speakers to campus, used 
luncheon gatherings, and scheduled special retreats (Gullat and Weaver, 1997). Several 
studies spoke about the effectiveness of the aforementioned activities, and included 
aspects of communication between the administration and faculty, and between a 
professional development committee, comprised of faculty representatives, and faculty, 
as whole. The effectiveness of the latter committee being mediated by the composition 
of the committee, that is, committees made up of recruited faculty members who were 
campus leaders were seen as more affective than committees constructed of volunteers 
only. 

Another intervening factor was found to be level of administrative support. Its impact 
was seen on both faculty acceptance and effectiveness of the programs. Often, the type 
of institution and the institutional culture impacted the expressed satisfaction and 
effectiveness of the development programs and activities (Sikes and Barret, 1976; 
Overlook, 1994). Missing from the previous research is assessment of the magnitude of 
the desire held by faculty for any particular type of program or initiative. Nor was a 
comparison made between what programs tenured faculty wanted, and the desires of the 
non-tenured faculty. 

The current study attempted to assess the professional development program for full- 
time faculty at a public teaching university, and create a hierarchy of the perceived 
usefulness of activities that have been implemented. A second goal of the study was to 
compare the differences between the tenured and non-tenured groups; and finally, a third 
goal was to initiate the construction of a scale to assess the factors underlying the 
faculty's responses. 



Method 

During the Fall 1999 semester, a questionnaire to elicit faculty input on the 
University's faculty-development activities was administered to the 240 full-time faculty. 
Sixty-seven usable surveys were returned, obtaining a response rate of 28%. The survey 
consisted of 37 items asking respondent opinion about activities, formats, 
communications and policies that are utilized in higher education. The respondent's task 
was to rate the usefulness, sufficiency, effectiveness, or satisfaction with a program, 
initiative or format that had been utilized in recent years. The levels of the descriptors 
were presented as five-point scales anchored at l=not useful, not sufficient, etc., to 
5=useful, etc. 

There was a third section asking about demographics: the number of development 
activities in which the respondent had participated in, in the past two years; number of 
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years teaching; number of years teaching at the University; highest degree and years 
since highest degree; tenure status; and department. 

The data were analyzed first in regard to percentages of responses received to the 
contingencies of the scales. Then, the data were put through a principal components 
(factor) analysis in an effort to discern the underlying dimensions. 

Demographics of respondents 

Sixty-two percent of the respondents were from the College of Arts and Science 
(n=34); 22% were from the College of Education (n=12); and, 16% were from 
Professional Studies (n=9). Twelve respondents did not provide this information. In 
looking at the institution as a whole, for the AY1 999-2000 the respondents from the 
Colleges of Professional Studies and Education were slightly over-represented and those 
from the College of Arts and Science under-represented (the institutional proportions 
were 71% Arts and Science, 17% Education, and 12% Professional Studies). 

Faculty with tenure were also somewhat under-represented because institutional data 
files reveal 75% of the full-time faculty were tenured. Of the total sample, 60% were 
faculty with tenure. 

Tenure status 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Tenured 


1 


40 


59.7 


67.8 


67.8 


Not tenured 


2 


19 


28.4 


32.2 


100.0 




• 


8 


'11.9 


Missing 






Total 


67 


100.0 


100.0 





Valid cases 59 Missing cases 8 

Other data about the respondents reveal that 75% had either a Ph.D. or Ed.D., and 
over 60% had earned that degree 10 years or more ago. 
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Years since earning their highest degree 











Valid 


Cum 


Value Label 




Frequency 


Percent 


Percent 


Percent 


Less than 10 


years 


22 


32.8 


37.9 


37.9 


10-15 years 




7 


10.4 


12.1 


50.0 


16-20 years 




10 


14.9 


17.2 


67.2 


21-25 years 




8 


11.9 


13 . 8 


81.0 


26-30 years 




2 


3.0 


3.4 


84.5 


31-35 




7 


10.4 


12.1 


96.6 


more than 3 5 


years 


2 


3.0 


3.4 


100.0 






9 


13.4 


Missing 






Total 


67 


100.0 


100.0 





Valid cases 58 Missing cases 9 

The average number of years the respondents had been teaching in higher education 
was 20 years; the median was 19.75 years, and the distribution was bi-modal at 10 and 30 
years. The range of years was from a minimum of 2 to a maximum of 39. 



The median number of years teaching at the present university was 9.5 years and the 
mode was 30 years (1 1 respondents reported 30 years). The reported range of years at 
the University was from 1 to 35. 



A profile of the respondent is that of a faculty member with tenure who obtained 
his/her highest degree over a decade ago, and has been teaching for approximately 20 
years, mostly at the present university; many earned their highest degree while teaching 
there. 

The Questionnaire Items 

The items asking about activities, formats, communications and policies required 
respondents to select the most appropriate choice from five-point, Likert scales. 
Depending on the wording of the question, the choices, ranged from, for example, "not 
useful" to "useful", or "not a need" to "very much a need". Similar rating scales were 
used with question about actual initiatives that occurred on campus. Descriptive statistics 
for these items are contained in the table below. They demonstrate faculty felt the most 
useful development activities were those that bring experts in the various disciplines to 
campus (Q5 and Q10), they were free to select their own professional development 
activities (Q22), and that activities to learn new classroom activities were useful (Ql). 
The respondents expressed a perceived insufficiency in the amount of money allocated 
for travel (Q23). 
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Descriptive statistics Q1-Q23 





N 


Mean 


Std. 

Deviation 


Valid 


Missing 


Q1 


63 


4 


3.79 


1.31 


Q2 


63 


4 


3.00 


1.34 


Q3 


64 


3 


3.72 


1.23 


Q4 


65 


2 


3.54 


1.34 


Q5 


65 


2 


4.23 


1.14 


Q6 


66 


1 


2.94 


1.43 


Q7 


66 


1 


3.32 


1.13 


Q8 


66 


1 


3.29 


1.31 


Q9 


65 


2 


3.11 


1.21 


Q10 


64 


3 


4.22 


.92 


Q11 


65 


2 


3.31 


1.36 


Q12 


66 


1 


3.77 


1.19 


Q13 


67 


0 


2.55 


1.25 


Q14 


66 


1 


2.65 


.97 


Q15 


58 


9 


3.50 


1.23 


Q16 


59 


8 


3.07 


1.31 


Q17 


58 


9 


2.69 


1.35 


Q18 


59 


8 


3.31 


1.42 


Q19 


58 


9 


3.00 


1.08 


Q20 


59 


8 


2.85 


1.23 


Q21 


59 


8 


2.51 


1.19 


Q22 


57 


10 


4.02 


1.01 


Q23 


59 


8 


1.95 


1.21 
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Descriptive statistics Q24-Q43 





N 


Mean 


Std. 

Deviati 

on 


Valid 


Missing 


Q24 


53 


14 


2.79 


1.29 


Q25 


56 


ll 


2.48 


1.32 


Q26 


51 


16 


3.43 


1.19 


Q27 


55 


12 


2.78 


1.26 


Q28 


60 


7 


3.35 


1.25 


Q29 


55 


12 


3.24 


1.29 


Q30 


59 


8 


2.69 


1.33 


Q31 


65 


2 


2.62 


1.33 


Q32 


65 


2 


2.83 


1.33 


Q33 


66 


1 


3.59 


1.08 


Q34 


66 


1 


2.44 


1.34 


Q35 


64 


3 


3.38 


1.24 


Q36 


66 


1 


3.32 


1.19 


Q37 


65 


2 


3.54 


1.17 


Q38 


50 


17 


3.26 


1.26 


Q39 


52 


15 


3.48 


1.16 


Q40 


48 


19 


3.46 


1.09 


Q41 


45 


22 


3.36 


1.38 


Q42 


41 


26 


3.07 


1.39 


Q43 


50 


17 


3.70 


1.23 



Extreme scores, or accumulations at the extremes can affect parametric descriptive 
statistics of Likert scales. It is often useful to assess percentages of responses received to 
the various contingencies of the scales. For the purposes of judging satisfaction with 
activities and formats, the two lowest choices were aggregated because it was felt the 
distinguishing qualities between the selection of a "1" and "2", signifying degrees of 
dissatisfaction, are often difficult to discern. For the same reason, the selections of "4" 
and "5" were aggregated. The middle point, i.e., "3", was left unchanged (complete 
frequency counts can be found in the appendix). 

The activities judged to be the most useful were those about the faculty members' 
discipline and classroom activities and instructional strategies. 



Useful Activities 


%responding 






Not 


%responding 




Useful 


Useful 


Classroom activities 


17.5 


66.7 


Assessments 


36.5 


36.5 


Instructional strategies 


17.2 


65.6 


Course content 


21.5 


58.5 


Research in discipline 


9.2 


81.5 
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BEST COPY AVAILABLE 



The format judged most useful was the one that brought an outside expert onto 
campus (which was found to be correlated with the desire for discipline-specific 
development), and it was also felt, more on-campus activities were needed at convenient 
times. The respondents said previous activities were scheduled at times that were not 
useful, but the on-campus retreats were helpful, nonetheless. They also felt inter- 
departmental meetings were useful, as were on-campus publications and University-wide 
workshops. 





%respondine 

Not 


%respondine 




useful 


Useful 


Useful Formats 

Departmental gatherings/meetings 


36.4 


39.4 


University-wide workshops 


19.7 


42.4 


Inter-departmental meetings/workshops 


24.2 


48.5 


Those held within each college 


27.7 


35.4 


Brought experts on particular topic 


6.3 


79.7 


Provide for on-campus publication or displays 


27.7 


47.7 




not 

needed 


Needed 


On campus development activities 


13.6 


62.1 




not 

sufficient 


sufficient 


On campus opportunities 


49.3 


25.4 




not 

convenient 


convenient 


Workshop times convenient 


37.9 


16.7 




not 


helpful 


Retreats with a focused, single theme 


helpful 

22.4 


58.6 



The results show faculty desire more on-campus activities centering on unique 
themes, the most helpful of which are centered around the faculty disciplines, particularly 
new research in those disciplines. 

The respondents do not appear too desirous for more activities concerning group 
dynamics. Approximately half of the respondents were neutral 12 to the question about 



12 This is implied by summing the percents finding it useful and not useful. The data can be found in the 
appendix, as well. 
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classroom group dynamics sessions, and most thought team building activities were not 
useful. 



Group Dynamics 



Classroom activities 
Team building activities 



%responding 

not %responding 

useful useful 

27.1 35.6 

43.1 27.6 



Most of the respondents said the primary source of on-campus encouragement came 
from within their own departments rather than from faculty members in other 
departments. This makes intuitive sense since these are the people with which most of 
the professional conversations would take place. 

Encouragement %responding 

not a %responding 

great deal great deal 

from members of my department 23.7 45.8 

from faculty in other departments 32.8 34.5 

Respondents were evenly distributed in their response to the question about 
sufficiency of university support for professional development, and they did not feel their 
input was actively solicited when campus-wide development activities were being 
planned. A clear majority did not feel pressured to participate in any activities, evidenced 
by the 70% who said they were free to select their own activities. An indicator of the 
perceived insufficiency of support is also found in the responses to an item about the 
university's travel allocation, where 73% said it was insufficient. 





%responding 






not 


%resDonding 




sufficient 


sufficient 


Support for professional development 


35.6 


33.9 




not 






actively 


actively 


Input solicited from University 


45.8 


22.0 




%responding 






not 


%resDonding 




free 


free 


Select own development activities 


7.0 


70.2 




not 






sufficient 


sufficient 


Traveling allocation 


72.9 


13.6 
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Several questions asked about university policies on development. It was felt the one 
concerning travel to conferences/workshops was unfair (reinforcing the feeling that there 
is insufficient support) and the one about release time was unfair; the one about 
sabbaticals was perceived as fair. Overall, the university was seen as concerned about 
faculty development, but could be more generous in funding individual initiatives. Three 
questions allude to this: one on tuition reimbursement, one on travel, and one on release 
time. A question asking about the policy governing distribution of funds received 
positive responses, overall, as did the question on sabbaticals. An interesting component 
of these data is that between 30-40% of the respondents were neutral to many of the 
questions. 

The majority of respondents felt their input was not sought when planning 
development activities, and aspects of the communication processes, themselves, were 
judged to be unsatisfactory. The respondents felt there is insufficient communication of 
changes in policy and ineffective processes of communication, and that information about 
other faculty members who could be seen as a resource for development was not 
effectively communicated. 



University's policy 


%responding 






not 


%resoondine 




sufficient 


sufficient 


Tuition reimbursement 


37.7 


28.3 




not 






fair 


fair 


Travel to conferences and workshops 


48.2 


21.4 


Sabbaticals 


17.6 


45.1 


Release-time 


36.4 


25.5 




not 






concerned 


concerned 


University demonstration for faculty development 


25.0 


50.0 




not 






fair 


fair 


Distributing funds for development 


27.3 


43.6 



Communication 


never 






asked 


asked 


Faculty input when planning development activities 


50.8 


30.5 





not 






sufficient 


sufficient 


Communication of changes in policy 


47.7 


26.2 
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not 






effective 


effective 


Communication processes, generally 


43.1 


32.3 


Communicating opportunities to participate in development 


12.2 


56.0 




does not 


does 




communicate 


communicate 


Communicating faculty as resource to campus community 


54.5 


21.2 





seldom 


frequently 


Faculty achievement recognition 


23.4 


48.4 




does not 


does 


Communicates achievements of others 


24.2 


43.9 




not 






able 


able 


Able to pursue development goals 


18.5 


58.5 



Past Campus Activities 

Several questions asked about previous initiatives sponsored by the university. 
Respondents were again asked to utilize a five-point scale with a " 1 " signifying low 
effectiveness of the activity/satisfaction with the activity, and "5" being high satisfaction 
or effectiveness. The initiative found to be most effective/"satisfying" was the Separately 
Budgeted Research and mini-grants. The least satisfactory was the Fall workshops 
introducing campus people. 



Low High 



SBR & Minigrants awarded to faculty 


22.0% 


62.0% 


Lunchtime presentations of Sabbatical, SBR, etc. 


18.8% 


56.3% 


Open, informal discussions with V.P. Carter 


26.7% 


46.7% 


Presentations of current research 


19.2% 


45.7% 


Full day April retreat 


28.0% 


44.0% 


Fall workshop introducing key campus people 


36.7% 


39.0% 



Twenty-two respondents, one third, said they had not been involved in any of the 
university-sponsored activities, mostly because they did not have time. 
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If you have not been involved in any of the above 
programs/activities/events , please tell us why. 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Not interested 


1 


3 


4.5 


13 . 6 


13.6 


Interested, no time 


2 


10 


14.9 


45.5 


59.1 


Did not hear of activities 


3 


1 


1.5 


4.5 


63 . 6 


Other 


4 


8 


11.9 


36.4 


100 . 0 




• 


45 


67.2 


Missing 






Total 


67 


100.0 


100 . 0 





Valid cases 22 Missing cases 45 

One question asked if the respondent felt there had been an improvement in the 
campus climate due to the development activities, to which 77% responded there had 
been at least some improvement. 

Do you feel that there has been an improvement in the campus climate in 
regard to faculty and professional staff morale and network 
opportunities as a result of faculty/staff development activities? 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


No improvement 


1 


14 


20.9 


23.0 


23 . 0 


Some improvement 


2 


34 


50.7 


55.7 


78.7 


Much improvement 


3 


13 


19.4 


21 . 3 


100.0 




• 


6 


9.0 


Missing 






Total 


67 


100.0 


100.0 





Valid cases 61 Missing cases 6 



The survey also contained a question asking how many development activities the 
person participated in, in the past two years. The modal response was four (29.8% 
responding this way) with six being the next highest number of activities (22.8%). 

Several categories into which respondents could be grouped were contained within 
the questionnaire. One obvious category is the dichotomous one of tenured vs. non- 
ten ured faculty. Forty of the respondents were faculty with tenure (59.7%), 19 were non- 
tenured (28.4) and eight ( 1 1 .9%) did not respond to that question. A table of descriptive 
statistics of the questionnaire items, which compares the responses of the two groups of 
faculty was constructed. It reveals subtle differences in their responses, generally a few 
tenths of a point— the largest difference being about nine tenths of a point, which was 
received to the question about the perceived utility of full-day on-campus faculty 
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workshops. The tenured faculty were more satisfied with this format than the non- 
tenured faculty (Q38). This mean difference was found to be statistically significant 
(t=2.21; df=42; p=.033) 13 . 

Statistically significant differences were found to responses to the question about 
tuition reimbursements, where the non-tenured faculty rated it more favorably then 
tenured (Q24) (t=2.45; df=3.72; p=.019); the question about fairness of sabbatical leaves 
(Q26), where the tenured faculty rated it more fair (t=2.67; df=32.78; p=. 012); the 
question of the helpfulness of on campus retreats about a single theme (Q15) with tenured 
faculty finding those retreats more helpful (t=2.07; df=26.8; p=.048); the question 
regarding ability to pursue one’s own development goals (Q37), again where tenured 
faculty rated this higher than non-tenured (t=2.206; df=28.72; p=.036); and, to the 
question about the University’s allocation of money for travel to conferences, where non- 
tenured faculty rated it more sufficient (t=-2.37; df=36.013; p=. 024). 14 

Factor analysis 

The first 43 questions of the questionnaire were put through an exploratory factor 
analysis with oblique (oblimin) rotation 15 (the other items captured demographic 
information). Rotation of the factor structure is a statistical mechanism for simplifying 
the emergent factors and aids in their interpretation. The interpretation itself is based on 
the unique contribution each item (i.e., question) makes to the emergent factors, that is, 
the magnitude of the items' loading (correlation) with the factor. This procedure 
identified 12 factors, which accounted for 75% of the total variance of these questions 16 . 
Of these factors, six were found to be stable, with stability judged tenable if four items 
correlated with the emergent factor (i.e., loaded on the factor) at equal to or greater than 
1.601. According to Stevens, (1996), stability can be judged with this criterion regardless 
of sample size, and factors can be interpreted utilizing items that loaded on the factors at 
levels greater than 1.361 for samples containing between 50 to 80 subjects. (The factor 
structure matrix is contained in the appendix.) 

Each factor, and a simplified name with which to interpret it, are contained in the 
following table, along with the respective questionnaire item numbers, statements, and 
loadings. (Only the stable factors are contained in the table). 



13 In reporting the following t-test values, the more conservative values, obtained by not assuming equality 
of variance viable, is used, even if the Leven's tests of equality was not statistically significant. This is 
because the sample sizes were very different, and a large number of tests were performed. It was felt this 
would attenuate the possibility of rejecting a true null hypothesis. 

The output for the Descriptive Statistics and Independent Samples Test may be obtained by contacting 
the author. 

An oblique rotation assumes the component factors are correlated. This contrasts with an orthogonal 
rotation (i.e., varimax), which assumes independence of the underlying factors. 

Each questionnaire item is a factor. The main concern of an exploratory factor analysis is identifying the 
factors to be retained, and the interpretation of the factors. Retention is based on the methodology 
suggested by Kaiser (1960), that is, the factor obtaining an eigen value greater than 1.0. An eigen value is a 
numeric value that consolidates the variance contained within a matrix, in this case the correlation matrix 
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Factor 1 Recognition and communication of faculty achievement 

Q18 I receive professional encouragement from members of my department. -.389 

Q27 University policy on release time is fair. -418 

Q28 The University demonstrates concern for faculty professional development. .509 

Q29 Fund distribution for development is fair. .384 

Q32 Policies are communicated effectively. .448 

Q33 The University communicates opportunities to participate in development activities. .422 

Q34 Faculty expertise is communicated to the community. .604 

Q35 Faculty achievement is recognized by the University. .857 

Q36 Faculty and staff achievements are communicated regularly by the University. .839 

Q38 Full day retreats improve development. .501 

Q41 Informal discussions with administration improves development. .629 

Q42 Participation in fall workshop introducing key people on campus improves development. .459 

Factor 2 Teaching and assessment 

Q1 The most useful activities are classroom activities. .830 

Q2 The most useful activities are centered around assessment. .427 

Q3 The most useful activities are instructional strategies. .890 

Q4 The most useful activities are concerned with course content. .607 

Q7 The most useful on-campus format is workshops/seminars. .362 

Q8 The most useful format is inter-departmental meetings/workshops. .605 

Q10 The most useful on-campus format is those that bring experts on campus. .408 

Q12 More on-campus development activities are needed. .385 

Q16 Workshops emphasizing classroom activities is useful. .463 

Q17 Workshops emphasizing teambuilding activities is useful. .378 

Factor 3 Campus and departmental meetings 

Q6 The most useful on-campus format is departmental meetings. .794 

Q10 The most useful on-campus format is those that bring experts on campus. -.624 

Q12 More on-campus development activities are needed. -.716 

Q13 There is sufficient on campus opportunities for professional development. .496 

Q18 I receive professional encouragement from members of my department. .596 

Factor 8 Policy on funding for long-term development activities 

Q22 I am free to select my development activities. -779 

Q24 The U.'s policy on tuition reimbursement is sufficient .403 

Q26 The University policy on sabbaticals is fair. 724 

Q27 University policy on release time is fair. -561 

Q28 The University demonstrates concern for faculty professional development. .652 

Q29 Fund distribution for development is fair. -659 

Q33 The University communicates opportunities to participate in development activities. .431 

Q35 Faculty achievement is recognized by the University. .366 

Q37 I have been able to pursue my professional development goals while at NJCU. .526 

Factor 9 Policies and funding for short-term development activities 

Q20 There is sufficient administrative support for prof, development -.534 

Q23 The amount of money the U. allocates for travel to conferences is sufficient -.875 

Q24 The policy on tuition reimbursement is sufficient -.616 

Q25 The policy on travel to conferences and workshops is fair -.903 

Q26 The policy on sabbaticals is fair -.437 

Q27 The policy on release time is fair -.597 

Q28 The U. demonstrates concern for faculty development -.573 

Q29 The U.'s policy on distributing funds for development is fair -.693 

Q30 The faculty are asked to provide input when development activities are planned -.374 
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Q35 The U. recognizes faculty achievement -.400 

Q36 The U. communicates achievements of faculty and staff regularly -.362 

Q41 Open informal discussions with VP of Academic Affairs (rating) -.455 

Factor 1 2 Planning and usefulness of previous activities 

Q7 The most useful format for on-campus development activities is University-wide -.471 

workshops/seminars. 

Q8 The most useful activities for on-campus development activities are in ter- departmental -.409 

meetin gs/ workshops. 

Q9 The most useful format for on-campus development activities is those held within each -.416 

department. 

Q1 1 The most useful format for on-campus development activities is those that provide for on- -.387 

campus publication or displays. 

Q15 On-campus retreats focused on a single them are helpful. -.534 

Q16 Workshops emphasizing classroom activities are useful. .683 

Q17 Workshops emphasizing teambuilding activities are useful. -.755 

Q21 Faculty input is solicited when planning development activities. -.771 

Q27 The U/s policy on release-time is fair -.543 

Q28 The U. demonstrates concern for faculty development -.412 

Q29 The U.'s policy on distributing funds for development is fair -.439 

Q30 Faculty input is solicited when planning development activities. -.81 1 

Q3 1 There is sufficient communication of changes in University policy. -.479 

Q32 Policies are communicated effectively. -.371 

Q34 Faculty expertise is communicated to the community. -.504 

Q41 Informal discussions with administration improves development. -.420 



The largest factor accounted for 24% of the variance and was mainly concerned with 
the University’s recognition and communication of faculty achievement. This factor 
demonstrates a decrease in perceived intradepartmental recognition as university-wide 
recognition increases. 

The second factor extracted was also judged to be stable, and it uniquely accounted 
for nine percent of the questionnaire’s variance. This factor was concerned with teaching 
and assessment. 

The third factor accounted for about eight percent of the questionnaire variance and 
was mainly concerned with on-campus departmental meetings. There were not a large 
number of items loading on this factor and it needs to be developed. But, it appears that 
with increasing numbers of departmental meetings, the desire for on-campus 
developmental activities decreases. 

The fourth factor judged to be stable was actually the eighth factor extracted. It 
accounted for about four percent of the questionnaire’s internal variance. It was called 
the factor on policy and funding long-term development activities. Items loading on this 
factor had to do with freedom to select one’s own development activities and the specific 
activities of release-time and sabbaticals. 

The fifth stable factor was the ninth one extracted. It accounted for three percent of 
the variance and was primarily concerned with satisfaction with policies and funding of 




102 



112 



short-term development activities. These activities include tuition reimbursement and 
travel to conferences. 

The sixth stable factor was the twelfth one extracted. It concerned the usefulness and 
effectiveness of on-campus development activities and communications. This factor 
accounted for approximately two and one half percent of the questionnaire variance. 

The oblique rotation allows the correlation among the factors to be assessed. Several 
were found to be moderately 17 correlated. The highest correlation was found between 
factors eight and nine r( 8> 9)=-.41— factor eight speaks about policies of long-term 
development and factor nine is about policies of short-term activities. The smallest 
correlation was between factors three and twelve, ^ 12 =. 001, “campus and departmental 
meetings” and “planning and usefulness of previous activities”. 



Component Correlation Matrix 



Component 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


1 


1.000 


.097 


-.080 


.220 


-.216 


.027 


.170 


.179 


-.286 


.038 


-.024 


-.323 


2 


.097 


1.000 


-.247 


.036 


-.119 


.120 


.180 


-.041 


-.026 


.039 


-.065 


-.341 


3 


-.080 


-.247 


1.000 


.082 


.059 


-.170 


.020 


-.027 


.010 


.033 


-.044 


.001 


4 


.220 


.036 


.082 


1.000 


-.073 


.139 


.192 


.124 


-.206 


-.119 


-.137 


-.145 


5 


-.216 


-.119 


.059 


-.073 


1.000 


-.040 


-.215 


-.199 


.198 


.014 


-.006 


.320 


6 


.027 


.120 


-.170 


.139 


-.040 


1.000 


-.018 


.058 


-.049 


.039 


-.004 


-.115 


7 


.170 


.180 


.020 


.192 


-.215 


-.018 


1.000 


.049 


-.178 


-.071 


-.075 


-.292 


8 


.179 


-.041 


-.027 


.124 


-.199 


.058 


.049 


1.000 


-.410 


.192 


-.057 


-.143 


9 


-.286 


-.026 


.010 


-.206 


.198 


-.049 


-.178 


-.410 


1.000 


-.175 


-.002 


.263 


10 


.038 


.039 


.033 


-.119 


.014 


.039 


-.071 


.192 


-.175 


1.000 


.068 


-.077 


11 


-.024 


-.065 


-.044 


-.137 


-.006 


-.004 


-.075 


-.057 


-.002 


.068 


1.000 


-.015 


12 


-.323 


-.341 


.001 


-.145 


.320 


-.115 


-.292 


-.143 


.263 


-.077 


-.015 


1.000 



Extraction Method: Principal Component Analysis. 
Rotation Method: Oblimin with Kaiser Normalization. 



The responses to items loading on the stable factors were summed creating scales, 
and the scale scores were aggregated for all respondents. Descriptive statistics for the 
aggregated scales of the questionnaire are below. They are presented for all respondents 
and separately for tenured and non-tenured faculty. Maximum scale scores varied in 
conjunction with the number of items in the scale, and higher maximums denote greater 
numbers of items comprising the individual factor. For example, "factor 1" 
(communication and recognition of faculty) contained 12 items and the maximum score 
is 60, which is based on the 5-point scales attached to each item. The average score on 
this factor was moderately high, 36.55, because the mid-point is 30. It is found that the 



17 Utilizing r=.30 as cited in Cohen and Cohen 1983 (using the formula t=r/square root (l-r 2 /n) correlations 
above +.247 were found to be statistically significant.) 
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highest score is on factor two— the factor dealing with teaching and, to a lesser degree, 
assessment (number of items=8; maximum possible score=40; obtained average=34.94). 
Factor eight received the next most positive rating, i.e., policies on long-term activities 
such as sabbaticals (seven items; maximum possible score=35; average obtained=30.41). 
The most neutral responses were obtained on factors three "campus and departmental 
meetings" and 12 "planning and usefulness of previous campus activities". 

A multiple analysis of variance (MANOVA) was performed on the scales' scores 
using tenure status as the grouping variable. No statistically significant differences were 
found between the groups' factor scores. Similarly, no significant correlations were 
detected between the factors and length of time teaching at the university. 

Frequencies: All Faculty 



Statistics 





Factor 


Factor 


Factor 


Factor 


Factor 


Factor 




1 


2 


3 


8 


9 


12 


N Valid 


31 


54 


56 


44 


34 


38 


Missing 


36 


13 


11 


23 


33 


29 


Mean 


36.55 


34.94 


16.70 


30.41 


36.00 


48.45 


Std. Deviation 


9.976 


7.398 


2.366 


7.267 


10.89 


11.90 


No. of items: 12 


8 


6 


7 


12 


16 




Highest possible score 


60 


40 


30 


35 


60 


80 



Factor 1 : Communication and recognition 

Factor 2: Teaching and assessment 

Factor 3: Campus and departmental meetings 

Factor 8: Policy on long term-development activities 

Factor 9: Policy on short-term development activities 

Factor 12: Planning and usefulness of previous activities 
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Frequencies: Tenured Faculty 



Statistics 







Factor 

1 


Factor 

2 


Factor 

3 


Factor 

8 


Factor 

9 


Factor 

12 


N 


Valid 


21 


35 


36 


30 


26 


27 




Missing 


19 


5 


4 


10 


14 


13 


Mean 




36.57 


35.69 


16.39 


30.30 


34.54 


48.44 


Std. Deviation 




10.59 


7.657 


2.429 


7.648 


11.20 


12.57 



No. of items: 12 8 6 7 12 16 

Highest possible score 60 40 30 35 60 80 

Factor 1 : Communication and recognition 

Factor 2: Teaching and assessment 

Factor 3: Campus and departmental meetings 

Factor 8: Policy on long term-development activities 

Factor 9: Policy on short-term development activities 

Factor 12: Planning and usefulness of previous activities 



Frequencies: Non-tenured Faculty 

Statistics 





Factor 


Factor 


Factor 


Factor 


Factor 


Factor 




1 


2 


3 


8 


9 


12 


N Valid 


8 


17 


17 


12 


7 


9 


Missing 


11 


2 


2 


7 


12 


10 


Mean 


33.50 


33.88 


17.06 


30.42 


38.14 


46.33 


Std. Deviation 


7.290 


6.800 


2.164 


4.582 


5.146 


10.37 


No. of items: 12 


8 


6 


7 


12 


16 




Highest possible score 


60 


40 


30 


35 


60 


80 



Factor 1 : Communication and recognition 
Factor 2: Teaching and assessment 
Factor 3: Campus and departmental meetings 
Factor 8: Policy on long term-development activities 
Factor 9: Policy on short-term development activities 
Factor 12: Planning and usefulness of previous activities 



Significance tests on the obtained means, when compared to the expectancy of the 
midpoint of the respective scales, reveals all of the obtained means are significantly 
higher than the expected average for each scale (remembering that z=1.96 at p=. 05). 



Means and Standard Errors 





Factor 1 


Factor 2 


Factor 3 


Factor 8 


Factor 9 


Factor 12 


Mean 


36.55 


34.94 


16.7 


30.41 


36 


48.45 


S.E. 


1.79 


1.01 


0.32 


1.10 


1.87 


1.93 


z score 


3.66 


14.84 


5.38 


11.78 


3.21 


3.08 



Discussion 

The sample was not truly representative of the university's full-time faculty in regard 
to college representation, however, the major divisions of Arts and Sciences, Education, 
and Professional Studies were represented in the appropriate order. That is, more 
respondents were from Arts and Sciences, the largest college, than from Education-the 
next largest, and Professional Studies-the smallest. The major discrepancy was the under 
representation of the College of Arts and Sciences, and the over-representation from the 
Colleges of Professional Studies and Education. There was also a slightly smaller 
percentage of tenured faculty who responded than exists on campus, but tenured faculty 
did outnumber the non-tenured. 

The responses revealed faculty felt more on-campus activities were needed, 
especially the type bringing experts in the academic disciplines to campus and those 
providing opportunities for publishing or displaying original work on campus. This 
could be the product of the greater number of tenured faculty responding because it was 
they, more than the non-tenured faculty, who expressed interest in these activities. The 
hierarchy of areas of interest list discipline specific presentations, classroom activities, 
and instructional strategies as the three highest priorities. There was interest in course 
content-specific programs, and little interest in programs showing assessment strategies, 
or team-work initiatives. 

The most desired format was that which brought experts to campus. Inter- 
departmental meetings were found to be useful, as were the on-campus publications, but 
to a lesser extent. Similarly, there was not great support for university-wide workshops or 
departmental gatherings, although the tenured faculty felt the university-wide workshops 
were more useful than did the non-tenured. 

There were no differences found between tenured and non-tenured faculty when 
asked about their perceptions of institutional support for professional development. 
Roughly equivalent numbers of both groups of faculty felt the institution did support the 
pursuit of professional development, and this type of perception has been found to be a 
motivating factor for individuals who have pursued development activities. Organizations 
that were supportive of professional development tended to have employees who 
participated at greater rates (Noe and Wilk, 1993). The majority of respondents felt they 
were encouraged most to pursue their professional goals from within their department 
and that they were able to select their own activities. According to Clark and Corcoran 
(1989) much of the developmental guidance a faculty member will receive comes from 
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within the department, whether it is on a professional or personal level. They continue, 
however, by saying that faculty in mid-career can develop “tunnel vision” if they have 
not had much contact with members of other departments. By saying inter-departmental 
meetings are useful, the respondents may be expressing a need for interaction with other 
departments’ members. 

Other areas in which differences between tenured and non-tenured faculty were found 
were in regard to sufficiency of tuition reimbursement, where non-tenured faculty rated it 
more sufficient, and sabbatical leave, and ability to pursue one’s personal goals, which 
were rated higher by tenured faculty members. 

When asked about the effectiveness of previous activities, the Separately Budgeted 
Research (SBR) and Mini grants were judged most effective by all faculty. The least 
effective was seen to be the lunch time presentations of sabbatical and SBR results. 
Unfortunately, most respondents did not feel their input was sought when development 
activities were planned. 

In analyzing the underlying factors, six were judged capable of supplying information 
about the survey responses, and no differences were found between the tenured faculty’s 
responses and those from non-tenured faculty. The administration's communication with 
faculty surfaced as the largest underlying factor. The average score on this factor was 
moderately high. This is interesting because items making up the factor revealed 
perceived insufficiencies— communication of opportunities, numbers of opportunities, and 
solicitation of faculty input when planning development activities. In a similar context, 
there was a perceived absence of campus wide communication of faculty expertise, 
although faculty achievements were communicated. This high score may be suggesting a 
need for greater communication, since a large part of what was being responded to in the 
survey was this communication factor, and the areas just mentioned are where the 
communication is lacking. 

Reinforcing the university’s mission as a teaching institution, the second factor, on 
development of teaching strategies, obtained the highest score. The factor about policies 
on long-term activities also received a high rating. The policies on short-term 
development were rated moderately high. The lowest levels of agreement were on the 
third and twelfth factors, which dealt with meetings and effectiveness of previous 
activities, respectively. The third factor, however, needs to be further developed, since 
there were few questionnaire items correlating with this factor. The addition of items 
would provide information concerning those aspects of meetings that were useful, and 
what to avoid in meetings whose agenda concern professional development. 

The twelfth factor contained all negative correlations. With the addition of different 
items, to provide positive loadings, and the deletion of some current items a clearer idea 
of the progress in obtaining faculty input into the planning process can be had. 





Future research should explore the qualities of the individuals that motivate them to 
participate in development activities. The psychological constructs know as the "Big 
Five" have been assessed in regard to leadership style (Judge and Bono, 2000) and 
service jobs (Hogan, Hogan, and Roberts 1996). It has been found in organizational 
settings that the desire to learn is a highly motivating factor in employees' motivation to 
participate in development activities, as is the perception of the support from managers 
and peers, and self-efficacy (Noe and Wilk, 1993). How those qualities relate to faculty 
in different life stages may supply information on developing initiatives for members at 
various career points. The present study found encouragement emanates mostly from 
within the department, but, again, the individuals' personality, education, or history, as it 
interacts with the departmental culture may be the factor spurring them to participate in 
particular types of activities. Considering the results of the current study, it may be asked 
if level of self-perceived self efficacy in the classroom, extent of current knowledge in 
the discipline, or personal level of psychological/cognitive development mediate the 
motivation towards participating in development activities in higher education. 

One thing lacking from the questionnaire utilized in the current study was the rate of 
participation and perception of the retirement planning activities. The questionnaire did 
not even address this as a form of development. In the future, inclusion of this type of 
content is recommended. 

Boice (1997) gives reasons why empirical research of faculty development programs 
has not often been utilized in program development. He says developers often see 
measurement as something that gets in the way of "something that is already working", or 
administrators perceiving that the money could be better spent elsewhere. Another 
reason is that measurement of efficacy in the classroom is a taboo subject to address. 
When it is brought up, it creates uncomfortable situations, such as learning empirically 
that one may be able to do something better than he/she is currently doing it. And, when 
behavioral interventions are included they are met with resistance because they entail 
collecting data on current practices and monitoring those practices for improvement. The 
current study does not offer a solution to the latter finding. It solely attempts to establish 
the groundwork for creating an empirical basis for assessing the magnitude of desire for 
types and formats of development programs, and for quantifying the extent of perceived 
support from peers and administrators, and measuring participants' satisfaction with the 
existing initiatives. It is hoped that the results will be used in constructing future 
programs and in performing research on faculty development. 

One area the current study did not address is funding of development activities. A 
future study should assess the effects of the utilization of funds allocated for development 
activities. Such a study can address the nature of activities the funds were spent on, the 
number of participants in on-campus and off-campus activities, what the participants 
brought back to campus/classroom/laboratory from the activity, usefulness for career or 
future planning, and overall satisfaction with the activity. This can provide insight into 
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the usefulness of the spending so that greater efficiency in the provision of professional 
and personal planning can be created. 18 
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Introduction 

Strategic planning - thoroughly understanding an institution’s strengths and 
weaknesses and carefully charting future directions - is vital to the effective management 
of colleges and universities. It also is integral to institutional change. As the American 
Council on Education observed in a 1998 report, “unplanned change is risky.” The 
current challenge to higher education is to chart intentionally a desired future congruent 
with our values and aspirations” (p. 3). Thus, strategic planning and change (or 
transformation) are intricately interwoven. Given the high levels of action they demand, 
they also represent higher education’s dynamic duo. Planning without transformation is 
unproductive. There is no purpose in planning if nothing changes and the resulting plan 
lies on a shelf to gather dust. Likewise, transformation without planning disregards the 
institution’s mission and often leads in distracting directions. 

Objectives 

This case study demonstrates the critical connection between strategic planning and 
institutional transformation. It traces the development of a strategic plan for a public 
baccalaureate institution and discusses how this strategic plan is linked to the 
transformation that has occurred on the campus over a four-year period. The study 
highlights numerous changes that have resulted from the plan. 

Institutional Background 

This analysis chronicles the dramatic institutional transformation of a small, four-year 
public college since 1996. Throughout its 163-year history. West Liberty State College 
in West Virginia's Northern Panhandle has served many first generation students, 
providing an affordable education with its solid curriculum and dedicated faculty and 
staff. The college enjoys a rich heritage as West Virginia’s oldest institution of higher 
education. It offers an associate degree program in dental hygiene as well as a full range 
of baccalaureate degree programs in the schools of liberal arts, science and mathematics, 
education, and business administration. The campus holds accreditation by the North 
Central Association and in the specialized disciplines of teacher education, nursing, 
dental hygiene, clinical laboratory sciences, and music. 
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By the mid-1990s, enrollment had dwindled from a high of 2,554 in 1981 to 2,412. 
While teaching and learning were taking place, there was no sense of energy or 
excitement, and the college had settled into a comfortable routine. The new president 
who arrived in July 1996 quickly recognized that this routine would not move the college 
toward a vibrant and successful future. In fact, he understood that the institution's failure 
to change and adapt could place its very existence in jeopardy. 

Challenges and Opportunities 

The state of the college at that time is aptly portrayed by the image of its brick and 
wrought iron front entrance gate. Portions of the wrought iron had separated and were 
awry. Struck by a vehicle during a snowstorm, the gate was left in disrepair for months. 
This transmitted a negative message: If the college did not care about its main entrance 
to campus, did it care about its internal operations? The campus certainly remained 
accessible; however, it lacked the focus on critical details that distinguishes a mediocre, 
sleepy institution from one that is animated, of high quality, and clearly focused on its 
future. 

In particular, the new president inherited a campus with no strategic plan, no master 
plan, no facilities plan, and no systematic budgeting process. An existing field house had 
recently been demolished as a result of severe structural deficiencies, limiting physical 
education and wellness opportunities for students. Other infrastructure problems caused 
by poor construction and deferred maintenance were mounting. Although approximately 
half of the student body resided on campus, student activities of all types were minimal, 
and the college was deserted on weekends. The institution offered few, if any, special 
academic programs for students such as an honors program or freshman experience 
course. While the concept of the freshman experience course had been discussed at 
length, it never moved beyond this stage to action. A program of student outcomes 
assessment was non-existent despite the accrediting requirements of the North Central , 
Association (and an upcoming visit scheduled for April 1998). Minimal computer 
technology was available for students and faculty; there was no infrastructure to support a 
campus-wide fiber optic network, and no computer labs had been installed. One of the 
college's premier academic programs — dental hygiene — was graduating excellent 
students despite the fact that the equipment in its clinic was twenty years old. 

In 1996 West Liberty State College had the highest percentage of tenured faculty in 
West Virginia public higher education. In addition, the average age of its faculty was 
among the highest in the state while the percentage of faculty with doctorates was the 
lowest, except for the community colleges. No provisions existed to reward faculty who 
displayed exceptional merit, and research and service were not considered important 
criteria in awarding promotion and tenure. Tenure did not involve a rigorous review and, 
in fact, was granted almost automatically. The college also had the highest ratio of FTE 
faculty per student in the state. Little or no ethnic diversity was evident among either 
faculty or staff. 
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Administratively, over twenty individuals reported directly to the president. 
Communication links between top administrators and the deans, department chairs, and 
division heads were very weak. Deans and department chairs were not asked to play an 
active role in managing the institution and met only infrequendy with the provost. No 
women served on the president’s cabinet. 

Although a foundation existed, the college had only a meager endowment of 
approximately $1.4 million despite its extensive history, and no special levels of donor 
recognition had been established to acknowledge major contributors. Furthermore, the 
state legislature had recently adopted a major bill requiring all state institutions to exhibit 
greater efficiencies over the next five years in order to qualify for any increase in state 
funding. At the same time, the institutions were asked to provide more responsive 
programming for their students and increase the level of compensation for faculty and 
staff. In short, the college was required to do more with less. Business as usual was no 
longer an option. 



The Campus Process: Strategies and Solutions 

According to ACE’s 1998 report. On Change: En Route to Transformation, 
“intentional change requires strategies and behaviors that are quite different from those 
associated with unplanned change. . .It involves charting a deliberate course” (p. 1). 
Given its situation in 1996, the college needed to embark on a clear course of immediate, 
transformational change to remain viable as the new millennium approached. 

Fortunately, the college enjoyed some strong positive forces that enabled it to tackle 
these challenges. First and foremost was the new president whose compelling vision 
inspired the campus and community; his passion to transform the institution aroused 
strong support from a critical core of faculty and staff who deeply believed in the college 
and were seeking far-reaching change. Several intensely loyal foundation and alumni 
board members also demonstrated a commitment to transforming the institution. The 
state's mandate to increase salaries through strategic planning only reinforced such vital 
support. 

Within two months of his arrival, the new president initiated a broad-based strategic 
planning process involving all constituencies. Thirty-five individuals, including faculty, 
staff, students, administrators, and key community leaders, participated in the strategic 
planning retreat and ultimately produced a plan that would set the college on a visionary 
and productive course. The resulting document outlined an ambitious agenda for 
advancing the college on several critical fronts: teaching and learning, technology, 
campus life, community outreach, reorganizing the college, and creating a student- 
centered campus. The twelve goals directly addressed the institution’s formidable 
challenges. 

Highly dedicated working groups intensely and systematically tackled each of these 
goals over the next several months. The initial strategic plan was completed and 



circulated to the campus for comment in early 1997. As the president acknowledged in 
his March 10, 1997, letter to the campus community, “To integrate the plan into the 
campus mainstream [now] requires every person to embrace the relevance and benefits of 
innovations recommended in the Vision to the Year 2000 Report. . .Our plan and its 
implementation must be a product of participation broad enough to cause ownership and 
result in specific decisions and actions to move the organization toward its future.” 
Through broad involvement of campus constituencies and constant communication, he 
had initiated the process that would engender this ownership. Not only did the president 
communicate the strategic vision to the campus, but he also conveyed this emerging spirit 
of enthusiasm and excitement to the institution’s statewide governing board during a 
meeting on the campus. Addressing board members, he portrayed the college as “a 
sleeping giant on the hill” who is about to awaken and make its presence felt. Alumni, 
the foundation board, business leaders, local public school superintendents, and state 
legislators all heard the same exhilarating message. 

The Beginning of Transformation 

Invigorated, faculty, staff, students, and administrators targeted action steps toward 
meeting the plan’s specified goals and objectives. At the close of the academic year, an 
annual update of accomplishments was compiled and shared with the campus 
community. During Fall 1997, the strategic plan was reviewed and updated, removing 
initiatives that were completed and adding new institutional priorities recommended by 
the faculty, staff, student, and administrator representatives participating in the planning 
retreat. The number of strategic goals was reduced to the seven that are currently in 
place: 



Goal One: Create a student-friendly environment by enhancing the student’s 
well-being. 

Goal Two: Establish a more challenging academic environment. 

Goal Three: Market WLSC as a high quality, affordable institution of higher 
education. 

Goal Four: Generate, maximize and wisely utilize sufficient financial resources 
to fulfill the mission and vision of the College. 

Goal Five: Develop and maintain a campus climate that promotes optimal 
employee performance, teamwork, continuous improvement and excellence. 

Goal Six: Have in place the technology and communication infrastructure to 
support the mission and core values of WLSC. 

Goal Seven: Extend WLSC into the community to meet continuously changing 
needs of our customers. 
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In October 1998, the president proudly stated to the campus, “Planning and action. . 
.are now a matter of daily operations. The collegial effort involved in creating the 
Annual Operational Plan represents a commitment to vision and planning, hours of hard 
work by many individuals, and dedication to action. . .Share it with your colleagues and 
be sure we hold one another accountable for its successful implementation." He also 
charged the college to “move forward with deliberate action steps to turn these objectives 
into achievements.” 

Since that time, the strategic plan has become an effective tool for keeping the 
campus apprised of priority activities and for building the momentum required to 
continue the institution's forward movement. In virtually every presentation to internal 
and external constituents, the president cites the strategic plan. During the Founder's Day 
2000 celebration, he observed that its vision "has helped us understand the challenges 
that are ahead and made us cognizant of the need to respond to need opportunities." This 
constant reference to the strategic plan, coupled with tangible results reflecting initiatives 
outlined in the plan, has made this document a highly effective mechanism for 
communicating progress at the institution. Deans and department chairs have been drawn 
increasingly into the college's decision-making process and are responsible for annually 
reporting progress on relevant initiatives in the strategic plan. During fall 1999, the 
president reviewed the recently updated strategic plan with deans and department chairs 
and then charged each department chair to discuss the plan with the faculty members in 
his or her area. The specific objectives identified in the plan also convey a very powerful 
message to political leaders and potential donors: "This institution is serious about 
planning and accountability, and it deserves your strong support." 

Based on the solid foundation articulated in the plan, the college has established an 
integrated planning process. The institutional budget plan is now directly linked to the 
strategic plan; through extensive budget hearings each spring, academic and 
administrative unit heads are called upon to justify their budget requests in relation to 
initiatives identified in the strategic plan. This increased level of involvement in planning 
and budgeting activities has heightened communication across campus and led to greater 
awareness of budget decisions. 

Over the past two years, West Liberty also has developed a ten-year campus master 
plan, a facilities plan, and a foundation plan that integrate with the strategic initiatives. 
Lending further coordination among these plans is the use of the same consultants to 
facilitate both the college’s annual strategic planning retreat and the foundation board 
planning process. The increased level of involvement in planning and budgeting 
activities has heightened communication across campus and led to greater awareness of 
budget decisions. As one department chair recently observed, “The strategic plan is a key 
document in driving the campus, including the budget and projects.” 

One of the major goals in the initial strategic plan concerned the need for 
administrative restructuring. Following deliberations with the deans and department 
chairs, this step was implemented in 1998. As a result, only eight positions (rather than 
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twenty) report directly to the president, and new hires brought three women to the 
cabinet. Seventeen academic departments were consolidated into ten. 



When the North Central Association visited the college in April 1998, the evaluators 
reported that “West Liberty State College has a mission statement that is well understood 
by students, faculty, professional staff, and support staff.” Their final report conveys a 
powerful sense of the exhilaration the team experienced at witnessing the tremendous 
changes that had occurred at the college in a short period of time. Highlights of their 
findings include: 

• The new president has brought a new sense of excitement, direction, 
professionalism, and impetus for change to an institution that was adrift for too 
many years. 

• West Liberty State College now has a new Strategic Plan that establishes goals 
and expectations of accountability at all levels. 

• The institution has faculty, staff, students, a Board of Directors, and alumni 
who are supportive of the spirit of change now present on campus. 

• Systematic efforts to reach out to the regional community through a number of 
initiatives such as the Science, Math and Research Technology (SMART) Center 
demonstrates the willingness of the college to be of service to its community. 



Continuing evidence of the systematic and highly visible effects of integrated 
strategic planning emerged at the College's September 2000 planning retreat. All 
participants were asked in advance to identify the College's top three to five 
accomplishments since the initiation of strategic planning in 1996. It is significant that 
the final list compiled from over fifty responses recognizes the strategic planning process 
itself as well as a clear focus on the plan, the budget review process, and the master plan. 
Among these "Top Ten" achievements are: 

1 . Campus beautification - Master Plan 

2. Construction of the new Academic, Sports and Recreation Complex 

3. Technology Expansion 

4. Focus on Students 

5. Increased Enrollment 

6. Strategic Planning/Budget Review Process/Enhanced Image 

7. Computer Labs/Legislative Support/Increased Accountability 

8. Clear Focus on Plan/Extemal Funding/New Department Structure 

9. Honors Dorm 

10. Leadership and Vision/New Dining Services/Progress in Assessment/ Marketing 



Evidence of Change 



Plan 
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Four years after the implementation of the strategic plan, the college is experiencing 
continued growth with its highest level of enrollment in nineteen years and the largest 
entering class since 1989. Students have acknowledged the new spirit; they are excited 
about the transformation they have witnessed, and some seniors who graduated last May 
expressed a desire to remain on campus for another year so they could enjoy further 
changes such as improved dining services and the $10.5 million Academic, Sports and 
Recreation Complex that was formally opened during a ribbon-cutting ceremony on 
Homecoming Weekend 2000. Student programs and activities have greatly expanded. 

An honors dormitory is filled with students, and the college is currently contemplating 
the creation of a second such residence hall for outstanding scholars. Faculty and 
department chairs have acknowledged responsibility for and assumed an active role in 
recruiting prospective students. 

Over $2 million has been dedicated to increasing faculty and staff salaries to more 
competitive levels over the past five years, and annual merit increases reward exceptional 
faculty initiative. New faculty hires are expected to hold a doctorate, and several faculty 
are currently completing doctoral programs. A creative severance plan offered several 
years ago enabled the college to review program staffing and to allocate its scarce 
resources more effectively. 

The president has established awards recognizing faculty excellence in teaching, 
research, and service. At a ceremony in March 2000, one faculty member expressed 
gratitude that research is no longer considered an “aberrant” activity at the college but, 
rather, an expectation. In addition, special presidential honors are accorded on rare 
occasions to employees or friends of West Liberty who have demonstrated extraordinary 
performance. 

Accompanying such activity is an increased emphasis on external research grant 
funding and a focus on development that has raised giving among alumni and friends of 
the college to new levels. Gifts to the college have increased by twenty-six percent or 
more in each of the past three years. A recent survey of donors attributed this support to 
“strong leadership” and the “sense of direction” provided by the strategic plan. A $1.87 
million grant from the National Science Foundation funds a center that provides hand-on 
science education to five county school districts of the region, serving 625 K-6 teachers 
and over 15,000 students a year. This is one of only five such projects in the United 
States. In 1999 the college received additional grants totaling approximately $1 million, 
including a $129,000 contribution which has enhanced music education through state-of- 
the art recording technology. The teacher education program is energized by a 
Professional Development School at one of the local elementary schools, one of only 
nineteen in the nation funded by Wallace Reader’s Digest. The state of West Virginia 
has recognized this new sense of excitement by selecting the college as the site for the 
Governor's School for the Arts beginning in 2001. West Liberty also received $185,000 
from the Governor toward renovation of the outdated dental hygiene clinic, allowing 
construction of a state-of-the-art facility. The college is further energized by a recent 
$100,000 federal grant designated for use in planning an innovative new center for 



instructional technology; this center will combine the institution's strong programs in 
science and mathematics education with those in communications, fine arts, and other 
disciplines to enhance instruction for undergraduates at West Liberty as well as for 
students in the public schools. A proposed new business information systems degree 
program, combining a solid background in information technology with business 
preparation, will benefit from this needed addition to the campus. The center also will 
offer professional development opportunities for public school teachers and will play a 
key role in the collaborative master's degree programs that the college is pursuing with 
area universities as a result of a new state statute. 

The campus is now wired to take advantage of technology, with fifteen computer labs 
available for student use. Grants from Verizon have extended Asynchronous Transfer 
Mode (ATM) connections to the campus as well as to the college's Warwood Center 
several miles away in Wheeling. Plans are underway to deliver college instruction to the 
region's high schools through video connections. Approximately thirty percent of the 
freshmen who entered in fall 2000 are enrolled in pilot sections of a new freshman 
experience course. A revised general education curriculum also was implemented this 
Fall, and several new specializations (including biotechnology and sports management) 
have been added to the curriculum. All academic departments have completed an 
assessment plan and are at various stages of refining and implementing techniques of 
measuring their goals. The first Faculty Symposium on Assessment held in October 2000 
highlighted these goals, and speakers representing each of the four schools described 
some of the innovative approaches used in their departments. 

In the midst of this widespread change, over 200 faculty and staff members sponsored 
a full-page advertisement in the local newspaper congratulating the college on its 
numerous accomplishments since 1996. Among the forty-six items cited were: 

• Renewed Commitment to Excellence through Long-Range Strategic Planning 

and Comprehensive Assessment; 

• New Faculty Evaluation and Merit Pay Plan; 

• Expanded and Revitalized Faculty Development Program; 

• Newly Opened Lines of Communication to and from Faculty Senate and Staff 

Council; 

• Restructured academic units; and 

• Commitment to a “Students First” Philosophy. 
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Conclusions 



As the president predicted to the Board of Directors in late 1996, it appears that the 
“sleeping giant” has, indeed, awakened, and is beginning to make its presence known in 
the local community and region. West Liberty State College has quietly and effectively 
made a difference in many lives over the past 163 years. The institution’s 
accomplishments and potential are just now beginning to be recognized more widely. As 
a result of the on-going strategic planning process, the college has begun to transform and 
re-invent itself to better serve a rapidly changing world. By focusing on the seven major 
goals that comprise our strategic plan, we created a campus culture in which our 
customers — students — receive highest priority. We restructured our finances and 
launched major efforts to improve an environment for teaching and research using the 
powerful tools of information technology. As we embark on the new millennium, the 
campus has become a dynamic community; frequent written and verbal communications 
acknowledge the strategic plan; and faculty, staff, and students are energized and actively 
working to accomplish the future directions we have helped envision for our institution. 

The strategic plan laid the foundation for the dramatic transformation that has 
occurred - and that is still occurring — by establishing a clearly articulated vision and 
much-needed direction for the college. The campus embraced the vision, gradually at 
first, but with increasing intensity as tangible outcomes were realized. The wrought iron 
and brick at the entrance gate have been repaired, and there is no turning back. As the 
American Council on Education report observes, change “is an ongoing, organic process 
in which one change triggers another, often in unexpected places. . .There is no point in 
time at which everyone can declare a victory and go back to ‘normal life.’” This 
statement is clearly evidenced in the new campus culture that has emerged at West 
Liberty State College. 
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Purpose of Research 

According to the Institute of International Education (Davis, 1998), 490,933 
international students were enrolled in U.S. colleges and universities during the 1998-99 
academic year, indicating a consistent increase over the last 40 years. It was also noted 
that over 1 1 percent of the total graduate enrollment across the country was comprised of 
international students. Clearly the student population in U.S. colleges and universities 
has become more diverse and this trend is also observed at the University at Albany, 

State University of New York, a mid-sized public research institution. 

Over the past decade, the international population has steadily grown in response to 
the university's continuous commitment to fostering the international dimensions of the 
campus (University at Albany's Strategic Planning Committee, 1998). It is reported that 
the total international student enrollment was 616 in 1990, and the year 2000 yielded 857 
international student enrollments, which is close to 6 percent of the total university 
student population (Office of International Education, 2000). Currently, 83 countries are 
represented among the international student population at Albany. 

International students are "non-immigrant" students who are authorized temporary 
visas for the duration of their full-time study in the U.S. and they must adhere to a 
number of strict federal rules and regulations which do not apply to U.S. citizens or 
permanent residents. International students are also individuals whose linguistic and 
cultural background are different from U.S. students. While the experiences of 
international students on campus might be similar to those of U.S. counterparts in some 
aspects, there are special needs among the international population, which must be 
addressed and served. To provide appropriate services to the international student 
population and facilitate a smooth transition from one culture to another, it is critical to 
monitor their needs and perform periodic needs assessment (e.g., Hammer, 1992; Lee et 
al., 1981; Selvadurai, 1991). 

In the coming years, an increase is expected in the international population at the 
University at Albany. Under these circumstances, it is vital for the university to know the 
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needs of international students and to examine whether or not our current services are 
satisfactory and meet international students' expectations. Additionally, as needs 
assessment is a continuous endeavor, selecting an effective research tool is essential. 
Nowadays, the Internet has been broadly used for college applications and registrations. 
Also, about 90 percent of college and university students in North America have ready 
internet access (Chidley, 1996; Terkla & Mcknight, 1998). The easy accessibility of the 
postage-free web-based survey may promote this type of research. 

In this paper, two issues were addressed. First, the perceptions of the international 
students about the serviced provided by Office of International Student Services (OISS) 
were examined. Second, web-based survey techniques were utilized in order to 
comprehend the strengths and weaknesses of this approach for possible future use. These 
issues are examined both in the literature and by this collaborative research project 
conducted by the Office of International Student Services (OISS) and the Office of 
Institutional Research (OIR) at the University at Albany in the Spring 2000 semester. In 
addition, we have learned that strong collaboration and communication among university 
units is a prerequisite for conducting a web-based survey research. This research project 
is an excellent example of inter-unit cooperation. 

Literature Review 

Needs Assessment and International Students 

Researchers have reported a variety of findings on international student needs 
assessment. Eid et al. (1989) surveyed the needs, satisfaction, and concerns of 85 
international students attending Eastern Oregon State College. With a response rate of 90 
percent, the 46-questions in seven different categories were analyzed by demographic 
variables to understand individual differences. The findings showed common needs and 
concerns that were also reported by international students in other colleges and 
universities in the U.S. International students felt their academic needs and interpersonal 
relationships with U.S. students were generally satisfactory; They wanted to develop 
more active interactions with the community; and sought more opportunity to improve 
their English speaking skills and to work on-campus. 

In another study. Hammer ( 1 992) conducted a "needs assessment" project for the 
Office of Intemational-Intercultural Student Services at the American University. A 
group of 231 graduate students (14 percent of the total international population) were 
interviewed and surveyed. The top needs were identified as follows: cultural variety in 
foods, employment opportunities, dealing with financial matters, and involvement of U.S. 
students in international activities. There were some overlaps in the findings of Hammer 
and Eid et al. (1989). 

A small-scale explanatory study by Luzzo et al. (1996) utilized an innovative method 
to determine the degree to which the needs of international students were being addressed 
by existing programs and services. During the last month of the fall semester, eight 
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undergraduate students answered a brief survey with 12 open-ended questions, and 
interacted with one another through a focus group interview. Interviews were videotaped 
to identify specific themes that emerged from the data. Their findings projected some of 
the findings in the previous studies: Overall, academic needs and interpersonal 
relationship needs were satisfied; Living in residence halls was a positive experience, but 
adequate variety in food was lacking. 

Studies by Lee et al. (1981) and Selvadurai (1991) discovered that the services 
typically provided to international students were both underutilized and perceived as 
ineffective. Lee et al. (1981) conducted a national scale study to determine the needs of 
international students. The sample of 1,900 international students from thirty U.S. 
colleges and universities with international student enrollments of over 300 was 
examined through a questionnaire organized along a number of categories (e.g., 
information needs, academic life heeds, linguistic needs). In every category, needs were 
not met according to the students' expectations and it was strongly suggested that U.S. 
institutions need to take a closer look at international students' needs and construct their 
programs accordingly. 

Selvadurai's 1991 study revealed the similar finding. The researcher evaluated the 
adequacy of selected academic and personal services to international students at New 
York City Technical College of the City University of New York. The response from 
1 37 students (response rate: 89 percent) to the 22-item questionnaire indicated the 
inadequacies in the overall services. The exception were the areas of personal services 
such as obtaining financial aid, counseling on immigration and tax matters, which 
attained minimum satisfactory levels. The researcher observed significant differences in 
the various opinions. As to academic services, male students were more satisfied than 
female counterparts; Oriental groups had more positive response than Middle 
Eastern/ Asian groups; Spanish/French speakers showed more favorable responses than 
Hindi/ Arabic and Chinese speaking students. In the personal service area, those who 
were proficient in reading English assessed services as adequate, while those with 
excellent reading skills assessed services as inadequate. Also, students with poor English 
speaking skills rated services as adequate, though excellent English speakers rated 
services as inadequate. 

Selvadurai (1991) pointed out that if different groups were chosen at different times, 
the evaluation of adequacy of services provided to international students might differ 
significantly. He suggested that the changing needs of international students at different 
points in time call for periodic needs assessment and following adjustments in services. 

Finally, Johnson (1993) examined the perceptions of international students at 
University of South Mississippi regarding the use and the effectiveness of services 
provided to international students. Seventeen international students were studied through 
Q-methodology, a type of factor analysis. The results drew three distinctive groups: 1) 
Dissatisfied non-users, 2) Selective users, and 3) Satisfied selective uses. There was no 
relationship between the length of time at the university and the use of the services. 





Johnson suggested future studies to find out which demographic characteristics were 
predictive of the use of student services. 

From this review, this area of research provides various and divergent findings as 
well as methods. This is partly because international students are so diverse in their 
distinctive cultural backgrounds. Harris (1996) discussed that a "cultural perspective" 
approach in needs assessment increases the effectiveness of services for students who are 
dissimilar in culture from the dominant culture since it can lead to a more precise 
identification of factors that influence students’ experiences and perceptions of the 
college environment. Additionally, there are arguments for the goodness of fit between 
qualitative research strategies and diverse population (Stabb, 1996). A mixed model of 
both quantitative and qualitative assessment may be desirable to get to the deeper 
individual perceptions as well as the broader numerical trends. The goals of this body of 
research have not been conceptually defined and the theoretical formulations have not 
been validated (Prieto, 1995). It is evident that this area of research is in a stage of 
development, leaving much room for continued work. 

Web-Based Survey Experiences 

Thus far, there is little higher education literature that discusses web-based survey 
experiences. A web-based survey was conducted by the Office of International Programs 
at Pennsylvania State University (Lynch & Wortman, 1999) to assess the needs of 
international students. This research showed that a web-based survey obtained a higher 
response rate compared with a mail survey conducted in 1997. This research also 
demonstrated that international students often asked for help from the International 
Student Office more with regard to practical needs (tax matters, travel documents, etc.) 
than for family or personal matters. This study did not attempt to analyze how differently 
international students interact with the International Student Office according to their 
different cultural backgrounds, nor did it address the pros and cons of their web-based 
survey approach. 

Several presentations at previous NEAIR conferences shared the technology of 
developing a web-based survey or suggestions for web-based survey research (Parrot & 
McKnight, 1998; Kelly, 1999; Palladino, 1999). These studies provided good examples 
of web-based survey design and administration as well as various technical resources in 
on-line survey design. However, they did not address other critical issues in the survey 
research process such as pilot testing, notification, confidentiality, and so forth. 

Therefore, more details on basic survey research issues and the complexities which need 
to be considered when conducting web-based survey approaches would serve 
Institutional Research practitioners well. 
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Research Method 

Subjects 

In the spring 2000 semester, there were 796 international students who were enrolled 
at University at Albany, State University of New York (Undergraduate: 144, Graduate: 
597, Non-degree/Exchange: 55). These students represented 82 countries. 66 percent of 
the total international population was from Asia, followed by students from Europe (22 
percent). North America (4 percent), Africa (2.6 percent). South America (2.3 percent). 
Middle East and Central America/Caribbean (1.5 percent), and Oceania (less than 1 
percent). They were in 56 different academic majors. Students in the Intensive English 
Language Program were not included in this study. It should be noted that University at 
Albany has three main campuses - uptown, downtown, and east campuses and OISS is 
located in the uptown campus. 

Instrument 

A 22-item questionnaire was developed. A mixed model was used to support the 
"cultural perspective" approach. The two parts were: 1) Quantitative: to rate selected 
services provided by OISS. There were five options - "very effective" - "neutral" - "not 
at all effective", and 2) Qualitative: to write comments and suggestions about specific 
services and overall services. The construction of the 22 items for the questionnaire was 
guided by the work of Fraenkel et al. (1996). The reliability of the instrument was 
examined by using previous reliable surveys as a model. The validity was established 
through a series of pilot studies and the review by the staff at the Office of International 
Education and OIR. In the pilot study, five international students from different 
countries, levels of study, and academic majors took the paper-and pencil questionnaire 
administered by a graduate assistant. Oral feedback was provided on simplicity of the 
language, clear meaning of the questions, and relevancy of the questions. Additionally, 
these students were asked whether they would feel different if completing same 
questionnaire on the Internet. 

The Survey Process 

In this section, the complete survey process will be described and followed by the 
design of the on-line survey. Different from a conventional survey, the questionnaire was 
released on the web for data collection and then followed by a mail survey distributed 
only to those who did not respond on line. The survey instrument was first 
conceptualized and questions were designed using pen and paper. Once all of the 
questions were finalized, OIR started the web-page design for the survey. 

The web-page was designed in a more vivid way to make the survey fun and 
intriguing for the respondents. Figure 1 shows the sample web pages of the survey. 
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Figure 1. The sample web pages of the on-line survey 





OIR employed an Active Server Page (ASP) 19 connected to two databases built in 
Microsoft Access to facilitate this on-line survey. One of the databases contained all of 
the international students' identification numbers as well as the date of their response. 

The other database was used to record survey responses. Students were required to key 
in their student identification number to successfully access the survey. Students who 
entered incorrect student identification numbers (ID's) were directed to a web page 
containing contact information for the survey administrator. The respondents' ID's were 
marked in the student-id database once the surveys were submitted and the date of 
response was automatically recorded in the database. This security system allowed us to 
verify that the respondents were from the target population and it also ensured that 
respondents could answer the questionnaire only once. The response date allowed the 
data manager to trace the trend of on-line responses over time. A "thank-you" page 
appeared right after a successful survey submission. The role of the OIR was not only to 
function as the data manager but also as data "guardian". Students were assured the 
confidentiality in a way which OISS could not see individual answers and comments with 
any identification. 

Another area of inter-unit cooperation was between OIR and Administrative Local 
Area Network (LAN) Services. The Local LAN Services set up the read/write access for 
OIR to be able to connect the survey and database with the university web server. A 
good historical working relationship and efficient communication between OIR and 
Administrative LAN Services expedited the processes of connecting the web-survey with 
the university web server. During the conversation between OIR and Administrative 
LAN Services, the survey's pilot testing was done on a personal server so that minor 
technical problems could be fixed before it was placed on the university web server. 

Once the survey and database were linked with the university web server, another 
pilot test was conducted to assure the connection between the survey and database on the 
university web server worked smoothly. As the technical details were being worked out, 

19 Special thanks to Jr-Ping Daniel Yang (Programmer Analyst, Research Foundation, SUNY) for his 
technical support with ASP scripts. 
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a traditional U.S. mail postcard was sent out to notify international students about this 
upcoming survey. OIR and OISS obtained international students' e-mail addresses from 
Academic Computing and updated this information from students' individual files in 
OISS for another notification, this time by e-mail. Once the survey and database were 
successfully connected with the university web server and the final pilot test was 
completed, OISS sent a cover letter to all the registered international students for the 
survey through e-mail. The cover letter message included the survey URL. 

A reminder was sent out on-line 10 days after the first e-mail. The last follow-up 
effort was the paper survey mailing to those who did not respond on-line. The responses 
submitted on-line was inserted into the Access database automatically, which allowed 
OIR to monitor the returns each day. The data collected via both on-line or paper survey 
were combined and analyzed after the posted survey deadline. 

Analysis and Results 

The survey response rate was 45.9 percent (365 out of 796). 69.9 percent of the 
respondents filled out their survey on-line. Figure 2 shows the trend of the web survey 
responses. The follow-up mail survey was responsible for 30.1 percent of the response. 
This survey effort compared very favorably with previous efforts to study the 
international student population at the University at Albany. A traditional mail survey for 
international students in 1996 that obtained 12.7 percent response rate (83 out of 655). 
Combining web-based survey techniques and traditional follow-up mail survey allowed 
us to maximize the response rate. The response rate of 45.9 percent, size of the 
respondent pool and the fact that respondent demographics (e.g., country of origin, 
gender, level of study, program of study, and age) largely mirrored the international 
student population, indicates a fairly high degree of confidence in the generalizability of 
the results. 



Figure 2. The trend of on-line responses 




Figure 3 compares the country of origin for the survey respondents with that of the 
international student population in the Spring 2000 semester. In addition, the 45.9 
percent response rate provided enough responses to produce a 95 percent confidence 
level with 3.78 confidence interval for interpreting the survey results. 

Figure 3. Demographic characteristics of the survey respondents vs. the 
international student population 





EUD Respondent 
Egga Population 



Overall, respondents were positive about the services provided by OISS. The mean 
response scores showed that student satisfaction with pre-arrival information, the 
international student electronic newsgroup, social opportunities (except coffee social 
hours and the end-of-year party), workshops, and general services offered by OISS fell 
between satisfied and very satisfied. The friendliness of OISS staff had the highest mean 
satisfaction score (4.67 on a scale of 1 to 5). The service rated lowest was the 
effectiveness of the orientation program, which was between neutral and satisfied with a 
mean of 3.88. The details for major findings in each service area are discussed below. 

Regarding the responses to the pre-arrival package, 78 percent of the respondents 
received the pre-arrival package and 55.7 percent of them were satisfied or very satisfied 
with the package, and another 23 percent expressed a neutral opinion. Little over 100 (30 
percent) of the respondents contacted OISS before their arrival and 90 percent of them 
reported that their pre-arrival questions were answered by OISS. The most frequently 
used methods of contact were e-mail and telephone. More housing information was most 
frequently suggested by graduate students and exchange students to improve the pre- 
arrival package. In addition, many international students expressed that they were eager 
to contact continuing students from the same country prior to their arrival. 

Sixty-three percent of the respondents participated the orientation programs. Asian 
students had a slightly higher absence rate than European students (37 percent vs. 25 
percent). Half (50.4 percent) of orientation participants rated the program as effective or 
highly effective in helping students adjust to the new environment, and another quarter 
(24.7 percent) were neutral. Comments on improving the orientation program showed 
that basic information such as housing, banking, how to get a driver's license, health 
insurance, social security number, and class registration procedures should be 
emphasized in the program. 

Electronic newsgroup listserv is for OISS to broadcast related news and information 
to international students. 58.6 percent of respondents subscribed to the electronic 
newsgroup and 81 .7 percent of them were positive about the effectiveness of the 
electronic newsgroup's function of keeping international students informed. A review of 
student comments suggested messages to be short and focused. 

Throughout the year, the OISS organized several social activities for international 
students. Just over half (51.2 percent) of the respondents attended at least one of the 
social events. Among the events. Thanksgiving dinner had the highest rating (4.46). 
Respondents were not as satisfied with both the coffee/tea social hours (mean=3.74) and 
the end-of-year party (mean=3.76) as with Thanksgiving dinner. Many students 
commented that they would like more intimate contact with OISS staff during social 
events. 

According to a crosstabs analysis, the number of undergraduate students who have 
attended events was lower than the expected number, while the number of exchange 
international students who had participated in events was higher than the expected 
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number. The expected number is the number of cases expected if the "student group" and 
"participation of event" are independent of each other. Table 1 shows the observed and 
expected numbers of participants broken by student degree levels. 

Table 1. Observed & Expected Number of Participants (Break by Student Degree Level) 



Participate? 


Count/Expected 


Undergraduate Graduate Exchange 


Yes 


Count 23 144 15 

Expected 31 141 9 


No 


Count 37 120 3 

Expected 28 124 8 



Comparison of the participation rate between the two largest international student 
groups, Asian (20%) and European (31%), it was found that Asian students were less 
interested in participating in the events than European students (see Table 2). 

Table 2: Observed & Expected Number of Participants (Asian & European Students) 



Participate? 


Count/Expected Asia Europe 


Yes 


Count 108 59 

Expected 115 49 


No 


Count 108 34 

Expected 101 43 



International student perceptions of OISS services can be characterized as mostly 
between satisfied and very satisfied. 85 percent of the respondents have visited OISS 
more than three times. According to a means test, undergraduate students tended to visit 
OISS less than graduate students did (mean = 4.18 vs. 4.87). There was not much 
difference between Asian and European students on the frequency of visiting OISS. In 
addition, the longer the students have stayed, the more often they visited OISS. 

Apart from general matters, students most frequently requested advisement on tax, 
immigration and employment matters. Graduate students requested advisement on tax 
matters more frequently than undergraduate students did, most likely as a result of 
teaching and research assistantship positions. Undergraduate students sought academic 
advisement more than graduate students did. Asian students asked for tax -related help 
from OISS more than expected while the number of European students who consulted 
tax -related advisement from OISS was less than expected. 

74.7 percent of the respondents positively rated the effectiveness of OISS to inform 
international students about federal regulations, and 18.4 percent had a neutral rating. 
Regarding the effectiveness of workshops, 88.8 percent of 167 workshop participants 
were positive about them. Student satisfaction ratings of services provided by OISS staff 
were all between satisfied and very satisfied. Respondents were very positive about the 
OISS staffs friendliness, phone courtesy, usefulness of advisor information, advisor 



O 

ERIC 



130 



139 



accessibility, sensitivity, effectiveness in solving problems, and in the time provided for 
discussion. In addition, respondents had a high degree of confidence in the information 
provided by OISS staff. 



Discussion 

The specific results of this investigation indicated that the majority of international 
students at University at Albany were satisfied with the services provided by OISS. At 
the same time, certain needs and concerns have been raised by many respondents which 
might not have otherwise been recognized (e.g., more comprehensive housing 
information, establishing a way to contact students from same country, modifications on 
orientation program). 

As a result of this study, OISS is developing a webpage which will contain useful and 
important information in more depth. Since the majority of international students used e- 
mail as the main tool to communicate with OISS prior to their departure, we expect that 
the webpage will be one of the critical mechanisms for OISS to disseminate information 
more thoroughly and effectively. Additionally, the webpage will show the results of this 
current study as a way of introducing our services to the public and our clients. 

There are several issues which are worthy of further investigation. For example, we 
need to learn more about what types of social events/programs appeal to students from 
Asia. Thus, it is our intention to conduct frequent mini-surveys or interviews on specific 
topics to identify particular needs according to different groupings of students. 

Moreover, this collaboration between the two offices contributed to the successful study 
and also raised more awareness of international students on campus. We envision 
continuing our synergistic efforts to promote quality services to the international 
population through on-going assessment. 

As for web-based survey techniques, looking into the whole survey process, an on- 
line survey did raise the response rate compared with the traditional mail survey 
conducted in 1996. After the experience of this web-based survey, several pros and cons 
were discovered. We found the on-line survey very accessible. Respondents could reach 
the survey with one click via e-mail as long as the e-mail recipient has their internet 
browser activated, which we believe most people do. In addition, a more interactively 
designed on-line survey can be easier and more fun for respondents to fill out than a 
paper survey. In this international student survey, respondents obtained contact 
information if they had trouble accessing the main survey with their student ID number. 
The connection between the on-line survey and response database allowed the data to be 
input automatically right after the respondents pressed the "submit" button. This 
mechanism minimizes the human errors that could be caused by manual data insertion. 

In addition, the on-line survey eliminates or minimizes the time needed for data insertion 
and survey mailing. Accordingly, administrative costs associated with data insertion and 
postage could be reduced. Lastly, the survey manager can monitor the survey returns and 
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have the most up-to-date data anytime. Through Access form and report design, data 
manager could even have running survey summaries available anytime upon request. 

One of the disadvantages of web-based surveys is the lack of flexibility it offers those 
survey targets that prefer anonymity. The ID validation mechanism may cause these 
people to refuse to participate. We do not see this issue as a major problem at the 
University at Albany since our students have historically been very willing to provide 
student ID numbers on surveys. In addition, web-based surveys are not viewed kindly by 
people who are not familiar with e-mail or internet technology. Another important 
consideration is that invalid e-mail addresses can become a critical issue in a web-based 
survey administration if potential respondents will be notified of this survey via e-mail. 

In this present study, approximate 70 percent of the response came from the on-line 
survey. We could have concluded data collection at this point with an acceptable 
confidence level. However, a follow-up mail survey was employed to include the 
opinions from those survey targets that were potentially not familiar with internet 
technology or missed the e-mail notification, or who just simply had not responded yet. 
As mentioned above, the follow-up paper survey did yield 30 percent of the responses. 
Therefore, if a maximized response rate is a critical issue for a survey, combining web- 
based survey with a traditional paper follow-up survey is a strategy we recommend. The 
URL of the on-line survey could also be included on the follow-up paper survey so that 
survey targets can choose their preference to fill out the survey. 

The web-based survey should not be viewed as a response rate "panacea" for any type 
of survey because not everybody appears to be familiar with and willing to use internet 
technology. However, a web-based survey could facilitate higher response rates in 
surveys targeted on college students because, as mentioned above, approximately 90 
percent of the college students in North America have ready internet access. However, 
when international students become the survey targets, one should consider whether or 
not their international students are familiar and comfortable with the internet technology. 
We found international students at the University at Albany to be most amenable to this 
approach. 

Post-survey communication between the survey host and the respondents is an 
important yet easily ignored stage in the need assessment process. It is crucial to inform 
the respondents that their opinions did matter. Therefore, in order to reinforce the 
connection and relationship between college students and the university service units, 
letting the respondents know how the service units are using the survey responses to 
improve service should be considered part of the survey process. 
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DEVELOPING AN ANALYSIS OF OUTCOMES FOR THE WRITING 
PROFICIENCY REQUIREMENT 

Kevin B. Murphy 
Institutional Research Analyst 
Office of Institutional Research and Policy Studies 
University of Massachusetts Boston 



Introduction 

Based on a user request, we have been involved with an ongoing analysis of the 
University of Massachusetts Boston Writing Proficiency Requirement (WPR). The 
University of Massachusetts Boston (UMB) is a public urban university with an 
extremely diverse student population that includes a high proportion of non-traditional 
students. The majority of our students enter as transfer students. The requirement consists 
of the successful completion of a timed essay examination, or the submission of a 
portfolio of work which includes several examples of papers written for courses and a 
new paper based on assigned readings and specific questions. It is designed to “assist 
students in acquiring critical skills. Foremost among these is the ability to present ideas 
clearly, correctly, and persuasively in English prose” (UMB Undergraduate Catalog). The 
requirement must be successfully completed as a prerequisite for graduation from the 
College of Arts and Sciences (CAS) and from the College of Nursing (CN). It is a high 
stakes requirement. There is no alternative path to graduation. Waivers are only granted 
to those who hold a bachelor’s degree from another institution and are entering UMB to 
acquire another bachelor’s degree. 

A number of courses have been designed to aid in the development of the critical 
skills needed for successful completion of the requirement. These are called Core or “C” 
courses. While these courses are offered in a number of disciplines throughout the CAS, 
they are generally overseen by the Core Curriculum Office which is also responsible for 
the administration of the WPR. There are also several courses specifically designed to 
prepare students who anticipate difficulty, or who have had difficulty meeting the 
requirement. It is these courses, two sequences of English composition courses, and a set 
of ESL courses that were the basis of the original research request. The original request 
that was made by the CAS Writing Proficiency Requirement Committee was basically a 
question that focused on the curriculum and its connection to success on the WPR. 

Re-formulating the Question 

This question assumes a view of the WPR as an event. The event has an outcome; a 
result of Pass or Retake. The question is about how another event, taking specific courses 
or the curriculum event, relates to the outcome of the WPR event. It was the wrong 
question. 
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The Writing Proficiency Requirement should be viewed as a process that begins 
before a single course is ever taken at UMB, rather than as an event. The better research 
question was about how the entire process (which includes the curriculum) contributes to 
success on the WPR. In order to analyze the outcomes of the WPR, we needed to first 
understand the entire process, and to identify stakeholders in other parts of the process 
that were beyond the focus of the Core Curriculum Office or the WPR Committee. This 
required a number of interviews and consultations that began in the Core Curriculum 
Office and branched out from there. There are well-established rules for the process. It 
was fairly easy to identify how the process works, or, at least, how it is supposed to work. 

The Process in Theory 

1 ) New students attend orientation and take the UMB English Placement Assessment 
(EPA) which is evaluated through the Freshman English Office. The ESL 
Program then further evaluates those with an ESL recommendation. 

2) The recommendations are entered into the computer system. 

3) University Advising (UA) accesses the EPA results, and the students are directed 
to the appropriate English courses. 

4) The students complete the recommended courses. 

5) All students complete the English Composition 101 and 102 sequence either at 
UMB, or bring it in as transfer credit. There is a UMB sequence of English 
Composition 101E and 102E that fulfills this requirement, and is specifically 
designed for non-native English speakers. 

6) Students who enter with fewer than 30 transfer credits must complete three 100 
level and two 200 level DcD courses in various departments. These courses are 
designed to focus on the reasoning and writing skills that are assessed by the 
WPR. Transfer students with 30 or more credits on entry are exempt from this 
requirement, but may also take these courses. 

7) The students attempt the WPR around the time they have accumulated 60 credits 
either by transfer or at UMB. They may do so by choosing either the Examination 
or the Portfolio option. 

8) Those who pass have no further requirement. 

9) Those who are required to retake the requirement enroll in NU250 if they are 
College of Nursing students. CAS students enroll in CRW Z282 if they intend to 
retake the requirement using the exam option, or CRW Z283 if they intend to 
retake the requirement using the portfolio option. Guidance is offered by the Core 
Curriculum Office which directs the WPR. 
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10) Additional tutoring and other forms of support are available through the Core 
Curriculum Office for those students who continue to have difficulty meeting the 
requirement until they have done so. Students continue to attempt the 
requirement until they receive a grade of Pass. 

The Process in Operation 

While it is necessary to identify how the process actually works, it is also necessary 
for the institutional researcher to identify the data that are available for each part of the 
process. We need to know about not only what is collected, but by whom, where it is 
stored, and how we may access it. This is important because it may be that the data that 
are needed to analyze the question are not currently available, and it should be a function 
of Institutional Research (IR) not only to recognize that, but also to identify ways to 
ensure that it becomes available for a future analysis. Therefore, as I describe how the 
process actually works. I’ll also discuss the data that are available at each step. 

On average, only about 85% of incoming students have attended orientation over the 
past ten years. The fall semester tends to have a higher attendance rate than the spring 
semester. University Advising keeps this information on a PC in its office. IR had no 
direct access to data concerning individual students. We only received the raw numbers 
to run against the admissions figures we maintain. 

The English Placement Assessment is much more important. The EPA is a holistic 
writing placement assessment. Students read several short passages, and write several 
paragraphs in answer to several questions. It is administered through the testing center at 
UA. It is evaluated by English Department faculty under the supervision of the Freshman 
English Program (FEP). The “score” is a recommendation for the student to take a 
particular course or sequence of courses in English. Until about three years ago, UA kept 
the results of the EPA on its own PC. At that time. University Information Systems (UIS) 
began keeping the data on a permanent student file. UIS is based on the UMass system’s 
main campus in Amherst MA, about ninety miles from our campus. Many of the systems 
they oversee tire slated to be phased out in the next several years as the University system 
converts its management systems from its existing mainframe environment. Resources 
for the existing system tire being diverted to create the new system. When UIS took over 
the EPA information, a miscommunication occurred so that three semesters’ worth of 
data were entered by UA staff without dates. They were under the impression that the 
system would insert the date. This currently makes the data impossible to locate by date, 
so it is difficult to identify the rates at which students took the EPA by semester. The 
information is eventually retrievable. Initially, it looked as though less than 5% of our 
students had taken the EPA over a period of several semesters. However, we were able to 
work backward from the group of students who had attempted the WPR, and by matching 
the records by student id number, get the EPA results. We found that over the past ten 
years, we had test results for only about 65% of the students. Several semesters had such 
a low rate that it seems certain that the data were somehow lost. Therefore, the analysis 
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will have to be focused on those semesters for which we have confidence in the data. We 
also found that a group that comprises about 10% of our fall admissions enters through a 
special program called Directions for Student Potential (DSP) that exempts them from the 
regular EPA. They are assessed through a different system, and the results are kept by 
their program on a separate system that is not readily available to IR. 

When the EPA has been evaluated by the English faculty, the student returns to 
University Advising, and the advisor reports the recommendation. However, when the 
English faculty makes the recommendations, they are partially based upon data self- 
reported by the students about their previous English experience. For example, a student 
who has already completed and received credit for the basic composition sequence but 
who needs additional work, would receive a recommendation for a course (ENG Z28 1) 
specifically designed for such students. When the adviser meets with the student, the 
student’s transcript should have been evaluated. At that point, the adviser may change the 
recommendation. If the example student did not actually receive prior composition credit, 
the adviser might change the recommendation to ENG 101 or ENG 102. The change in 
the recommendation is not collected anywhere. This makes it difficult to determine 
whether the student has complied with the recommendation. This is important because 
compliance is not mandatory. We can only determine the value of this step of the process 
if we know which students utilized it. 

Because neither completion of the EPA nor compliance with the recommendation is 
mandatory, a number of students self-register for courses. The Freshman English 
Program has developed a shadow system to deal with this. On the first day of all of the 
English composition classes, the instructor administers a mini-EPA. All of the students 
are asked to read a short passage and to write a response to several short questions. The 
Director of the FEP assembles a small task force from among the English Dept, faculty to 
assess this informal instrument by the beginning of the next class session. It is used to 
provide a placement recommendation for those students who avoided the formal EPA, 
and to confirm proper placement for those who completed it. No documentation of any 
kind exists for this system. No data are gathered on the results. As with other 
recommendations, this recommendation is not binding, and the student may insist on 
remaining in the course for which s/he registered. It is likely that this part of the process 
has a significant impact, because a number of students eventually register for specialized 
classes for which they have had no formal recommendation. 

The ESL staff from Academic Support Services assists in evaluating both the formal 
EPA and the informal mini-EPA. They also conduct assessments and work closely with 
non-native English speakers who have self-identified or been referred to their office at 
any time. They have the results of assessments they complete outside of the formal EPA 
process. However, this information is stored in a database in their office and is not readily 
available to IR. 

The “C” course requirement is fairly straightforward by rule. The students who enter 
with fewer than thirty (30) credits must complete it. However, they don’t necessarily 
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have to complete it before they attempt the WPR. In practice, most students do not 
complete the five courses before attempting the WPR for the first time. Because of the 
credit cutoff, and the numbers of transfer students we enroll, it is difficult to easily 
identify all of the students who are subject to the requirement. However, among our own 
first time freshmen who are all subject to the rule, less than 30% completed the five 
courses. In fact, over 35% had completed only two or fewer of the courses. If the student 
successfully completes the WPR, s/he will often attempt to get the rest of the courses 
waived. Because the number of these courses is limited, and they are designed to prepare 
the student for the WPR, the waiver is often granted. Other students attempt to receive a 
composition waiver by attempting the WPR before completing the English Composition 
sequence. This waiver is almost never granted. All course data is stored with, and 
controlled by, UIS. 

There are several courses offered that are generally understood to be for students who 
have not passed the WPR on a first or subsequent attempt. However, these courses are 
sometimes taken before the first WPR attempt by students who anticipate extreme 
difficulty. Data on these are available on the course file. For students who have 
attempted the WPR several times without passing, individual tutoring is offered. This is 
coordinated through the Core Curriculum Office in concert with Academic Support 
Services. Information concerning tutoring is kept on a separate system by Academic 
Support, and it is not available to IR. 



Other Issues 

Prior to the June 1996 WPR exam, a single WPR record was kept for each student. 
This record held data regarding the student’s most recent attempt. For those with multiple 
attempts, no information was available about previous attempts. This meant that we could 
not analyze students’ behavior between attempts, because we didn’t know when the 
previous attempt occurred. After an earlier attempt to analyze outcomes on the WPR, the 
system was changed to accommodate records for multiple attempts. UIS changed the 
input programs. For, this reason, our analysis was to include only those students who 
attempted the WPR for the first time in June 1996 or later. When we first began to access 
data on the WPR last spring, we noticed that we had more than one record listed as the 
first attempt for a number of students. There is a field called “noattmpt” that should but 
does not always identify the number of the attempt. The true identifying field is called 
“examtype”. However, we found that even when the two fields agreed that it was a first 
record, we occasionally had a previous record for the student. In order to identify the 
correct members of our group, we eventually settled for a set of conditions. If both of the 
fields agreed that it was first attempt and it was the first record we had for the student, we 
selected the record for the first attempters data set. 

Once we had our initial data set, we also noticed unexpected values in several fields 
on a number of records. The Core Curriculum Office is responsible for data entry for the 
WPR results. I contacted the clerk who normally enters the data to ask for a key to the 
values. There isn’t a written one. She was taught how to enter the results by the person 
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who had the job before her. She thought that the unexpected values were probably 
entered by a temporary worker while she was on leave. The values entered in some of the 
fields were valid, but for other fields. No documentation exists on this campus. 

Similarly, when we accessed the EPA data, the results field seemed to be filled with 
garbage characters. The Testing Center finally provided an old sheet of the codes they 
had been using when they controlled the data, with additional UIS codes penciled in. The 
new codes consisted of punctuation marks. No other documentation exists on our 
campus. 



Implications for IR 

The Stakeholders 

The WPR Committee and the Core Curriculum Office initiated the study. While they 
recognize that University Advising, the Freshman English Program, and Academic 
Support Services with the ESL Office all play a part in the WPR system, they tend to 
view the WPR as an event or pairing of events. Students take courses. Students attempt 
the WPR. The other stakeholders are sometimes viewed as adversaries rather than as 
compatriots. For example, the WPR Director suspects that a number of advisers don’t 
pass along the EPA recommendations to their students because they don’t believe in their 
value. The other stakeholders occasionally share a suspicious view of the Core 
Curriculum Office and the entire WPR process. Recently, we held a meeting about the 
progress of the study. It was called by the Director of the WPR. Nobody from any of the 
other stakeholders’ groups was invited. 

In instances like these, it may be that IR can bridge the gap between stakeholders. 
They may be suspicious of each other. They may in fact have conflicting goals. This 
sometimes makes it difficult for them to communicate effectively with each other. I 
found that each of the people I interviewed in the various departments had a very good 
picture of the idiosyncrasies of their particular part of the process, and was happy to talk 
to me about it. However, each also assumed that they knew how the other parts of the 
process worked, because the other parts would, of course, work according to the existing 
rules. At one point, I asked the Director of Freshman English how many of our students 
took their English composition courses at other schools. She replied that there shouldn’t 
be any because it would be against the rules. The rules require prior permission for our 
students to be able to take off campus courses and then to transfer them into UMB. The 
Registrar’s Office here acknowledged that that was the rule, but that it was in place in 
order to deal with unusual courses. Basic English composition would be readily accepted 
from any other accredited institution. Rules are made to be broken. 

It can be the job of IR to analyze quite a number of the processes on campus. We have 
to learn the rules and how they are applied. That can put us in an excellent position to 
facilitate communication and understanding if we are trusted by the various stakeholders. 
They have to trust that we will tell their parts of the story as accurately as we can. 
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The Data 



Our office does not control or maintain most of our data. As I’ve noted, some of it is 
kept in informal shadow systems on PCs in offices around the campus. Most of it resides 
with UIS in Amherst. UIS is particularly short of resources, and has many other clients 
and demands for its attention. Two of the major data problems that occurred happened 
when UIS changed or took over an existing system. Our office was aware of at least one 
of those changes. We should have run at least a small-scale test on the data sets shortly 
after the systems were changed. We might have found the attempt numbering problem in 
the WPR results data set before four years of data were entered, and perhaps identified 
the missing date values on the EPA before three semesters had to be corrected. However, 
we didn’t. In the future, we should. We are responsible for the integrity of the data we 
report. Often, we are responsible for the extra work to correct the problems with the data. 

The data can also give us information about the process. The very small numbers of 
ESL recommendations in the EPA file led me to suspect that ESL students were being 
evaluated differently. It was so. They receive a different code on their EPA records that 
was not shown on the coding sheet that I was given by the Testing Center. Their records 
have now been accessed. The same was true of the DSP students. I only learned that they 
were exempt from the regular process when I found that they had very few records in the 
EPA file. I was then able to ask the right questions of the responsible people. 

Communicating What Is Possible and What Is Meaningful 

We recently participated in a meeting with members of the WPR Committee and 
several other interested parties. I found that a number of them had unrealistic 
expectations. One professor wants correlations for the various courses and success on the 
WPR. That is quite possible. The answer is that completing the courses designed for 
people who enter needing extra work on their English skills is negatively correlated with 
success on the WPR. However, it’s also a meaningless answer. The proper question is 
how much taking such a course changes the probability of success for a student who 
needs to develop those skills. This is why the EPA recommendations are so important. It 
establishes a baseline for our analysis. Communicating this is difficult, but it is necessary. 

We need to be able to communicate this need for a baseline in order to persuade 
people to do the extra work to capture data for us. The informal mini-EPA is a good 
example. It is an outstanding system. The extra work done by the English faculty to 
assure that each student who is taking a composition course is in an appropriate class is 
remarkable. Probably the last thing they want do is the additional work of formally 
tracking those recommendations. We also need UA to perform the extra work of entering 
any changes in the EPA recommendations that they make. Without that extra work, we 
can never assess just how valuable their efforts might be. 




141 



150 



Conclusion 



The pass rate on the initial attempt at the WPR is about 80%. The rate on subsequent 
attempts is about the same. Eventually, sometimes with a great deal of support, almost 
everyone succeeds. The challenge is to develop an assessment of the process given the 
process as it actually operates. Enough data probably exists in usable form to produce 
reasonable results now. We can probably even answer the question that was originally 
asked. However, we need to prepare to do a better job in the future. Part of that is helping 
to formulate the question so that the answer will be meaningful. 
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Abstract 

The research described in this paper 21 consists of a detailed analysis of the 1995 
National Household Education Survey (NHES:95) Adult Education Component in light 
of the findings and recommendations of the Commission for a Nation of Lifelong 
Learners (CNLL). Findings in this paper include variations in self-reported barriers and 
motivations for participation by socioeconomic status, age, gender, ethnicity, industry of 
employment, and types of courses taken. Demographic differences were also found 
between those who participate in credential programs, personal development courses, and 
work-related courses. 



Adult Education in the 1990s 

Adult education has often been described as being on the fringe of the higher 
education landscape (Maehl, 2000). The vast majority of educational institutions seem to 
focus their attention on recruiting and retaining traditional-aged students, despite the fact 
that between 1985 and 1995 the number of adult students enrolling in higher education 
grew more rapidly than did the number of traditional-aged students (Snyder, Hoffman, & 
Geddes, 1998). 

In the 1990s, the Commission for a Nation of Lifelong Learners (CNLL) was 
assembled with a grant from the W. K. Kellogg Foundation. CNLL developed a series of 
recommendations, implementation strategies, and policy implications based on its 
findings. The Commission recommended that there be broad acknowledgement of the 
link between universal lifelong learning and America's position in the global economy, 
that access to lifelong learning resources be made equitable, that new technologies be 
effectively used to deliver adult education, that there be a reorganization of the delivery 
of adult education, and that adult education and lifelong learning be given funding in 
proportion to their significance for America’s future. 



20 Roy Gunnarsson is presently employed at Regents College. 

21 For a copy of the full version of this paper, please e-mail the first author at mnesler@regents.edu 



Motivations and Barriers in Adult Education 



One line of research on motivations has examined the influence of demographic 
characteristics on participation in adult education. Fujita-Starck (1996) further suggested 
that demographic characteristics alone are not sufficient to identify motivations. Instead, 
she suggested that adult learners be grouped by curricula (personal development, 
professional enhancement, and the arts). Fujita-Starck found that those enrolled in 
professional enhancement courses had professional motivations, while adults enrolled in 
personal development courses were motivated by improving communications skills. 
Those enrolled in arts courses were motivated by the desire for social contact. However, 
Scanlan & Darkenwald (1990) concluded that research into motivational factors alone 
has not been sufficient to distinguish between adult education participants and 
nonparticipants. Motivational concerns can be interrelated to the logistical problems and 
situations that occur in adult life. 

Recent research has measured both perceptions of barriers and motivations for 
participating in adult education. For example, in comparing participants and 
nonparticipants in continuing education courses, Henry and Basile (1994) found that 
major changes in the person’s life created barriers to enrollment. Cost was also cited as a 
major deterrent by nonparticipants. It should be noted, however that the term “barrier,” 
referring to some absolute blockage, is being replaced in the adult education literature by 
the term “deterrent.” The latter term reflects something more dynamic that is working in 
combination with other forces (Valentine and Darkenwald, 1990 as cited in Silva, 
Cahalan, & Lacireno-Paquet, 1998). 



The Current Study 

Several papers have been generated using the NHES:95 data (Bills, 1998a, 1998b, 
1998c, 1999; Hollenbeck, 1999; Kim, Collins, & McArthur, 1997; Kim, Collins, & 
Stowe, 1997a, 1997b; Kim, Collins, Stowe, & Chandler, 1995; McArthur, 1998), as well 
as technical guides to using the data (Brick & Broene, 1997; Collins, Brick, Kim, & 
Gilmore, 1996; Collins & Chandler, 1996; Nolin, Collins, & Brick 1997). 

The current research was designed to explore previously unanswered questions using 
the NHES:95 data, primarily in light of the work of the Commission for a Nation of 
Lifelong Learners and the research on participation in adult education programs. The 
questions addressed include: 

1) Do the self-reported barriers and motivations for participation vary by 
demographic characteristics such as socioeconomic status, age, gender, 
ethnicity, industry of employment, in addition to the types of courses taken? 

2) What are the major demographic differences among those who participate in 
credential programs, personal development courses, and work-related courses? 
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3) Who are the major providers of adult education? Are there demographic 
differences in who is attracted to different providers of adult education? 

Method 



Sample 

A total of 19,722 individuals completed the adult education component of NHES:95. 
Of these, 1 1,713 were AE participants and 8,009 were non-participants. In order to 
provide accurate information for important subgroups of the population, oversampling 
was used for subgroups. 

Analysis 

In the present analyses, the WesVarPC software was used to produce weighted 
population estimates, standard errors, and subsequently for statistical tests. WesVarPC 
uses a replication method to estimate standard errors (Brick, Broene, James, & 

Severynse, 1997). Crosstabulation cells with frequencies lower than 30 were not included 
in the analyses, except as where noted. 

Results 

All percentages reported are within-group percentages for each particular category. 

All reported differences are statistically significant at the p<.05 or pc.Ol level. The 
Bonferroni correction for familywise error rates was applied to all multiple comparisons. 

Barriers to work-related courses 

The greatest barrier reported to taking work-related courses was time (46.9%) 
followed by money and costs (29.7%). There was substantial variation in reporting each 
different barrier, however. A great deal of the variance in the different barriers was 
accounted for by age, gender, and income. 

Age . Younger individuals were less likely than older individuals to report time as 
the main barrier to work-related courses. For example, both individuals aged 16 
through 24 and individuals aged 25 through 34 were less likely (40.1% and 41.3%, 
respectively) than individuals aged 35 through 44 (52.6%) to report this barrier. 
However, the two younger (16-34) age groups and the older (35+) age groups did not 
differ appreciably among themselves in reporting this barrier. 

Whereas younger people tended to be less likely than older age groups to report 
time as the main obstacle, they were more likely report costs as the main barrier to 
work-related courses. Again, both individuals between 16 and 24 years of age and 
individuals between 25 and 34 were more likely (37.8% and 34.9%, respectively) 





than the 35 through 44 age group (24.9%) to report money and costs as the main 
barrier. 

Gender . Gender differences were found in three barriers to work-related courses. 
First, men were found to be more likely (54.2%) than women (40.7%) to report time 
as the main barrier. Second, women were more likely (32.3%) than men (26.6%) to 
report cost as the main barrier. Third, women were more likely (1 1.3%) than men 22 
(2.5%) to report child care as the main barrier. 

Ethnicity . There were a few ethnicity differences in reporting barriers. African 
Americans were least likely (33.0%) to report time as the main barrier to taking work- 
related courses. Both Caucasians (49.5%) and ‘Other’ ethnicities (50.4%) were 
statistically more likely than African Americans to report this. 

Socioeconomic status . The general trend was that individuals with higher income 
were more likely to report time as the main barrier. Thus, individuals in the highest 
income category were more likely (59.6%) than individuals in the middle category 
(45.6%) to report this. Respondents with incomes between $20,000 and $40,000 were 
in turn more likely than individuals with incomes below $20,000 (29.0%) to report 
time as the main barrier. The pattern for reporting cost as the main barrier to work- 
related courses was opposite that of reporting time as the main barrier. Individuals 
with annual household incomes below $20,000 were more likely (43.3%) than 
individuals with incomes between $20,000 and $40,000 (31.5%) to report money and 
costs as the main barrier. Individuals in the $20,000-$40,000 income group were in 
turn more likely than individuals with annual incomes above $40,000 (19.5%) to 
report cost as the main barrier. 

Motivations for Participation 

Statistics on motivations for participation are reported in Table 1. 

Credential courses 



The main reasons for taking credential courses was to train for a new job or career 
followed by improving, keeping up, or advancing in one’s current job. 

Age . Younger individuals were less likely than older individuals to take credential 
courses to improve in a current job but were instead more likely to take such courses 
to train for a new job or career. 

Socioeconomic status . Individuals with lower household incomes were less likely 
than individuals with higher incomes to take credential courses to improve in their 



22 The unweighted cell frequency for men in this test was 29, one less than the suggested inclusion 
frequency of 30. 
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current jobs. This pattern was reversed when taking courses to train for a new job or 
career. Lower income individuals were more likely than higher income individuals to 
take credential courses to train for a new job. Further, middle income individuals 
were more likely than high income individuals to take credential courses to train for a 
new job. 

Personal development courses 

Gender . Men were more likely than women to take courses in this category to 
improve in one’s job. Women, on the other hand, were more likely than men to take 
such courses for personal, family or social reasons. 

Work-related courses 



Age . The youngest age group differed from all the older age groups (except for 
retirement age) in their reasons for taking work-related courses. Individuals ages lb- 
24 were less likely than individuals ages 25-34, 35-44, 45-54, and 55-64 to take such 
courses to improve, keep up, or advance in their current jobs. Individuals in the 
youngest age group were more likely than individuals ages 25-34, 35-44, and 45-54 
to take the courses to train for a new career. 

Socioeconomic status . The last important source of variation in reasons for taking 
work-related courses was income. Overall, the higher an individuals income, the more 
likely was that individual to take work-related courses to improve, keep up, or 
advance in his or her current job. Conversely, high-income individuals were less 
likely than middle income and low income individuals to take work-related courses to 
train for a new job or career. No statistical difference was found between low and 
middle income individuals after correction for familywise error rate. 

Demographic Profiles of AE Participants 

The demographic data are here reported within each demographic variable in order to 
highlight the differences between the different types of courses. See table 2 for actual 
percentages. 

Age . The majority of participants in credential courses are young. Participation in 
credential courses seems to decrease rapidly with age. The age distribution is more 
even for personal development and work-related courses. 

Gender . More women than men were enrolled in credential courses. The same 
trend is present and more pronounced in personal development courses. However, 
participants in work-related courses seem to be slightly more likely to be men than 
women. 





Ethnicity . The largest ethnicity constituency for all three course types was 
Caucasian. Based on the population constituencies, Caucasians seem overrepresented 
except for credential courses. African Americans seem underrepresented in work- 
related courses. Hispanic students seem underrepresented in all three types of courses, 
but especially in work-related courses. 

Educational attainment . Almost half (48.7%) of the adult population belong to the 
lowest two educational categories. These individuals have not had any formal 
education beyond high school. This large group is underrepresented in all three types 
of adult education. On the other hand, individuals with college degrees (Associates 
degree. Bachelors degree, or Postbaccalaureate degree) were overrepresented in all 
three types of adult education. 

Socioeconomic status . Participants in credential courses were proportionally 
distributed (compared to population constituencies) across household income 
categories. However, participation in personal development courses and in work 
related courses was less proportionally distributed. In both cases, lower income 
individuals tended to be underrepresented and higher income individuals tended to be 
overrepresented. 

Provider Statistics 

Credential courses 



Postsecondary institutions provided most (90.1%) of the credential courses taken by 
the survey respondents. Because of the high degree of uniformity, no statistical tests were 
performed for providers of credential courses. 

Personal development courses 

There was a great deal of variability in providers of other structured courses. The 
most common providers of these courses were churches or other religious organizations 
(28.5%), private or community organizations (8.5%), tutors or private instructors (5.3%), 
or some other organizations (2.0%). These providers were aggregated into the 
‘Miscellaneous’ category, which subsequently accounted for 44.3% of these courses. 
Other main providers of these courses were postsecondary institutions and business or 
industry (20.3% and 18.8%, respectively). However, there were a few exceptions to this 
overall distribution of providers. 

Age . For the youngest age group (16 through 24), the most common provider of 
personal development courses was postsecondary institutions (42.9%). Miscellaneous 
providers and business or industry were listed as the provider by 30.7% and 10.8%, 
respectively, by these respondents. 
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Work-related courses 



Over half (5 1 .9%) of the work-related courses were provided by business and 
industry and nearly a fourth (24.2%) by postsecondary institutions. However, a few 
demographic differences were found. 

Gender . Overall, men were more likely (58.0%) than women (46.0%) to take 
work-related courses from business or industry providers (r=7.35, pc.Ol). 

Ethnicity . There also appeared to be ethnicity differences in business or industry 
as the provider. However, the apparent ethnicity difference was found to be qualified 
by gender. That is, an ethnicity by gender interaction was found. For Caucasian, 
African American, and Hispanic men, there were no differences in reporting business 
or industry as the provider for work -related courses (58.6%, 57.0%, and 55.9%, 
respectively). Caucasian women were more likely (48.2%) than both African 
American women (34.5%; t= 3.85, p<.01) and Hispanic women (37.1%; r=2.47, 
p<.05) to report business and industry as the provider. There was no statistically or 
practically significant difference between African American and Hispanic women. 

Discussion 

The NHES:95 data includes questions about perceived barriers adults faced, but these 
questions were only posed to individuals who had an interest in a work-related course and 
knew of such a course they wanted to take but could not. While there is obvious logic in 
this approach to asking questions about barriers, it may underrepresent the actual barriers 
individuals face, in particular, barriers or deterrents to taking credential and other types of 
courses. Individuals who had less specific knowledge about courses they wished to take 
would also be excluded from the data collection, as only those people who knew of a 
course they wanted to take were asked to describe the barriers to participation. 

The most important barriers to adult education are time and cost. Time seems to be 
the greatest barrier, especially to older workers, men, and higher income individuals. For 
younger workers, time and cost are equally deterring to adult education, and for 
individuals with lower incomes, cost is the most deterring factor to adult education. 
Unfortunately, because barrier information was only collected for work-related courses, 
no comparison between types of courses can be made. The type of courses was, however, 
related to the reason individuals reported for taking a particular course. 

In terms of motivations, research has shown that age and gender covary with 
motivations to participate in adult education (e.g., Morstain & Smart, 1974). In addition, 
motivations have also been found to vary by the types of courses people take (Fujita- 
Starck, 1996). The data analyzed here largely support these findings. Motivations were 
found to vary by course type (personal development, credential, or work-related courses), 
as well as by demographic characteristics. 
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The two overall most important reasons to take credential courses were both job- 
related. Most important was training for a new job or career, followed by improving, 
keeping up, or advancing in one’s current job. Some individuals were more likely than 
others to take credential courses to train for a new job. Not surprisingly, younger 
individuals and individuals with lower incomes were more likely to seek out new careers 
by taking credential courses, perhaps in order to increase their earning power. Older 
individuals and individuals with higher incomes, on the other hand, are more likely to 
seek to improve in their current jobs. Perhaps these individuals are satisfied with their 
career choices and simply seek to advance within their careers. One interesting finding 
was that the two reasons, training for a new career and improving in one’s current job 
were reversed in importance for individuals belonging to an ethnic minority. That is, for 
minority members, the most important reason for taking credential courses was to 
improve in one’s current job, followed by training for a new job or career. 

The most common reason for participation in personal development courses was for 
personal, family, or social reasons. However, a significant minority of individuals were 
taking these courses for work-related reasons. As would be expected, these individuals 
were more likely to be employed than otherwise. They were also more likely to be men 
rather than women. 

Most individuals take work-related courses to improve, keep up, or advance in their 
current jobs. This reason was the most common even for unemployed individuals who 
sought to improve in their previous fields of employment. 

Demographic characteristics and differences were also assessed for the different 
course types. Most noteworthy were differences in age, ethnicity, and educational level. 
Participants in credential courses tended to be younger than participants in other types of 
courses; almost half of the credential students were younger than 25 years. On the other 
hand, participants in personal development courses seemed more evenly distributed 
across age groups. Participants in work-related courses were mostly of mid-career age 
with very few young and old participants. 

Ethnicity was another source of differences in course constituencies. For all types of 
courses, Caucasians made up the largest ethnic group. However, when compared to 
population constituencies, it was found that Caucasians were slightly overrepresented in 
personal development courses and work-related courses, but not in credential courses. 
African Americans were slightly overrepresented in credential courses but 
underrepresented in personal development and work-related courses. Individuals of 
Hispanic origin were underrepresented in all three types of courses. 

Educational attainment differed widely between participants and non-participants. 
Almost half the adult U.S. population was exceedingly underrepresented in adult 
education. These were individuals with no more than a high school diploma. On the other 
hand, individuals with college degrees were greatly overrepresented in adult education. 
The differences in educational attainment between participants and non-participants also 




15QL59 



varied with the type of courses. The overrepresentation of individuals with at least some 
college education would be expected for participation in credential courses. After all, 
higher education is a sequential process where students must attain one degree before 
continuing to the next. However, college-educated individuals were most overrepresented 
in work-related courses and least overrepresented in credential courses. 

The different course types also differed widely in who provided them. Credential 
courses were almost exclusively provided by postsecondary institutions. Personal 
development courses, on the other hand, were provided by a wide range of organizations. 
Most commonly, personal development courses were provided by churches and religious 
organizations. However, postsecondary institutions, business or industry, and private or 
community organizations also provided a significant portion of these courses. There were 
few trends in the types of providers of personal development courses. Most notably, for 
the youngest age group, 16-24 years, postsecondary institutions were the most common 
type of provider of personal development courses. 

About half of the work-related courses were provided by business or industry. 

Another fourth of the courses was provided by postsecondary institutions. One issue of 
interest regarding work-related courses is whether access to the courses is equitable. This 
issue is especially important when the courses are provided by business or industry since 
such courses are often mandated and/or sponsored by the employer. Our analysis shows 
that, overall, men are more likely to participate in work-related courses provided by 
business or industry than are women. Further, among women, there was a relatively large 
gap in participation rates between Caucasians and members of minority groups. Among 
men, however, there were no ethnicity differences in participation rates for work-related 
courses provided by business and industry. 

The data reported here serve to both expand our understanding of some of the issues 
surrounding participation in adult education and serves to confirm.findings from the 
literature on adult education. One of the important recommendations suggested by the 
Commission for a Nation of Lifelong Learners was that equity of access to adult 
education be achieved. The data provided by NHES indicate that as a society, we still 
have some progress to make on achieving this goal. 
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Table 1: Reasons for participation 

Improve, keep up, or Train for new job or Improve basic skills Meet requirements for Personal, family, or 
advance in current job career diploma or degree social reason 

CR PD WR CR PD WR CR PD WR CR PD WR CR PD WR 
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Table 2: Demographics of adult education participants 





Credential 

/ % 


Personal 
Development 
/ % 


Work-Related 

/ % 


Population Total 
/ % 


Age 


16-24 


9,682 


47.4 


4,821 


12.8 


3,287 


8.3 


22,439 


11.8 


25-34 


5,648 


27.7 


8,948 


23.8 


10,413 


26.2 


40,326 


21.3 


35-44 


3,428 


16.8 


9,650 


25.6 


12,728 


32.1 


42,304 


22.3 


45-54 


1,475 


7.2 


6,509 


17.3 


9,462 


23.8 


31,807 


16.8 


55-64 


159 


0.8 


3,550 


9.4 


3,109 


7.8 


21,824 


1 1.5 


65-99 


20 


0.1 


4,161 


11.1 


697 


1.8 


30,876 


16.3 


Gender 


Male 


9,192 


45.0 


14,276 


37.9 


19,653 


49.5 


90,275 


47.6 


Female 


11,220 


55.0 


23,363 


62.1 


20,042 


50.5 


99,301 


52.4 


Race 


White 


15,138 


74.2 


30,079 


79.9 


32,999 


83.1 


144,602 


76.3 


Black 


2,573 


12.6 


3,927 


10.4 


3,371 


8.5 


20,808 


11.0 


Hispanic 


1,377 


6.7 


2,159 


5.7 


1,851 


4.7 


15,705 


8.3 


Other 


1,325 


6.5 


1,474 


3.9 


1,474 


3.7 


8,461 


4.5 


Highest Grade Completed 


Up to 1 1th grade 


265 


1.3 


3,086 


8.2 


1,860 


4.7 


36,385 


19.2 


High school 


2,280 


11.2 


8,798 


23.4 


7,918 


19.9 


55,919 


29.5 


Vocational/technical school 


244 


1.2 


1,337 


3.6 


1,383 


3.5 


6,327 


3.3 


Some college 


9,779 


47.9 


8,715 


23.2 


7,686 


19.4 


34,435 


18.2 


Associates degree 


1,727 


8.5 


2,730 


7.3 


3,202 


8.1 


9,975 


5.3 


Bachelors degree 


3,271 


16.0 


7,254 


19.3 


9,698 


24.4 


26,858 


14.2 


Postbaccalaureate degree 


2,845 


13.9 


5,718 


15.2 


7,949 


20.0 


19,677 


10.4 


Labor Force Status 


Employed 


14,358 


70.3 


25,936 


68.9 


36,622 


92.3 


117,833 


62.2 


Unemployed 


1,415 


6.9 


1,419 


3.8 


906 


2.3 


8,167 


4.3 


Not in labor force 


4,639 


22.7 


10,284 


27.3 


2,167 


5.5 


63,576 


33.5 


Industry 

Agriculture 


187 


0.9 


609 


1.6 


660 


1.7 


3,792 


2.0 


Construction 


441 


2.2 


1,157 


3.1 


1,304 


3.3 


7,320 


3.9 


Manufacturing 


1,493 


7.3 


3,085 


8.2 


4,446 


11.2 


19,808 


10.4 


Transportation 


813 


4.0 


1,983 


5.3 


2,661 


6.7 


8,441 


4.5 


Retail & Wholesale 


4,178 


20.5 


3,948 


10.5 


3,073 


7.7 


22,568 


11.9 


Finance 


1,016 


5.0 


1,795 


4.8 


3,559 


9.0 


7,506 


4.0 


Service 


7,872 


38.6 


12,977 


34.5 


17,290 


43.6 


48,027 


25.3 


Government 


1,068 


5.2 


2,092 


5.6 


3,557 


9.0 


7,843 


4.1 


Misc. industries 


608 


3.0 


1,328 


3.5 


1,998 


5.0 


6,593 


3.5 


N/A 


2,735 


13.4 


8,665 


23.0 


1,147 


2.9 


57,677 


30.4 


Household Income 


$0 - $20,000 


6,233 


30.5 


7,885 


20.9 


4,586 


11.6 


56,853 


30.0 


$20,001 - $40,000 


5,905 


28.9 


11,800 


31.4 


10,947 


27.6 


58,839 


31.0 


Over $40,000 


8,274 


40.5 


17,954 


47.7 


24,162 


60.9 


73,883 


39.0 


Total 


20,412 


100.0 


37,639 


100.0 


39,695 


100.0 


189,576 


100.0 



Frequencies are weighted population estimates and are reported in thousands. 
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CURRICULUM REVIEW AT A VIRTUAL UNIVERSITY: AN EXTERNAL 

FACULTY PANEL APPROACH 

Mitchell S. Nesler, Director of Research, Academic Programs 

Regents College 

Amanda M. Maynard, 23 Research Associate 
Regents College 



Originally founded in 1971 by the New York State Board of Regents as the External 
Degree Program of The University of the State of New York, Regents College is a 
currently a private, independently chartered institution based in Albany, New York. It is 
governed by a board of trustees comprised of a national group of prominent leaders in 
education, business, and the professions. On January 1, 2001 Regents College will 
change its name to Excelsior College, although the college’s mission will remain the 
same. The mission of the college is to help remove barriers that exist for working adults 
in their quest for higher education while still maintaining rigorous standards of academic 
excellence in its external degree programs. Since it’s inception, more than 90,000 
individuals have earned accredited associate and baccalaureate degrees in business, 
liberal arts, nursing, and technology from this unique college. Approximately 15 percent 
of the students enrolled in Regents College come from New York State; the remaining 85 
percent come from all other states and several foreign countries. All of the college’s 
enrolled students (approximately 17,000) are at a distance. To ensure academic 
excellence, the college utilizes multiple methods and measures to assess program 
effectiveness. Graduate follow-up surveys, employer and/or supervisor surveys of 
graduates' work, and external faculty review of curriculum and program outcomes are 
just some of the measures of program effectiveness instituted by the College. 

Regents College does not have a resident faculty, just as it does not have resident 
students. Each degree program (business, liberal arts, nursing and technology) has a 
faculty committee that is responsible for overseeing its respective degree programs. The 
approximately 350 faculty of Regents College are drawn from many colleges and 
universities as well as from industry and health care facilities. They establish and 
monitor academic policies and standards, determine degree requirements and the ways in 
which credit can be earned, develop the content for all examinations, review the records 
of students to verify their degree requirement completion, and recommend degree 
conferral to the Board of Trustees. 

Review of the curricular structure is a challenging task for any college or university, 
but poses additional challenges for virtual universities. Regents College offers external 
degree programs in 18 concentrations within Liberal Arts. The faculty and administration 
were interested evaluating the curriculum structure for each of these concentrations in 



23 Amanda M. Maynard is currently an Assistant Professor of Psychology at Bard College. 



terms of both strengths and weaknesses. The overarching goal of the reviews was 
program improvement, the documentation of the curriculum’s equivalence to traditional 
four-year institutions, and an evaluation of the currency of the curriculum structure as 
compared to traditional four-year institutions. 

By their structure, external degree programs offer the student flexibility to obtain 
credit toward a degree from a variety of sources, including courses taken at accredited 
traditional institutions. Regents College also offers direct assessment of student learning 
through a suite of proficiency exams, developed by the college’s Assessment Unit. The 
selection of comparison institutions becomes a challenging task for the virtual university. 
The selection of comparison institutions for traditional institutions may focus on 
institutions of identical affiliation, student body size, entrance requirements, and 
geographic location. Virtual universities, however, serve students without such 
geographic boundaries. Regents College, in particular, serves traditionally 
underrepresented groups in higher education, does not have entrance requirements, and 
serves students from around the globe. One of the first challenges for conducting a 
curriculum review was selecting comparison institutions. 

A sensible approach is to conduct the curriculum review with the overarching goal of 
the review in mind during the design phase. The complexity of reviewing 18 
concentrations within Liberal Arts was considered along with the nature and mission of 
the college. To lend some consistency to the review process, it was decided to select 
comparison institutions that would remain constant across each of the 18 reviews. Using 
a fixed set of institutions alleviates the potential for a given curriculum to fair well due to 
any particular characteristics of the institutions selected for a particular review. In 
addition, this practice allows for some comparisons in terms of the review of outcomes 
across programs. As the goal was to investigate the equivalence and currency of the 
Regents College curriculum as compared to those of traditional institutions, only four- 
year institutions having majors identical in name each of the Regents College 
concentrations were selected as comparison institutions. From the set of traditional four- 
year institutions having majors identical to the Regents College concentrations, the final 
set of ten institutions were selected varying in institution size, affiliation, and geographic 
location. The resulting set of institutions included institutions whose self-reported 
entrance difficulty for admission was reported as “moderately difficult,” and whose 
geographic location varied with the intention of selecting institutions representative of 
programs nationally and ensuring a rigorous review process. 

The Regents College Biology concentration was the first curriculum to undergo 
review. The remainder of this paper discusses the procedure of the review. Further 
outcomes of the curriculum review for Biology are discussed as well as the strengths and 
weaknesses of the procedure utilized. 
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Method 

Participants 

Again, the overall goal of the review was to ensure equivalence and currency of the 
Regents College curricula to that of traditional four-year institutions nationally; thus, the 
selection of external evaluators was approached with some of the same criteria used to 
select comparison institutions. Criteria used for selection of external faculty included 
extensive teaching experience in Biology, current affiliation with a four-year institution, 
and an openness to the notion of distance education and the mission of Regents College. 
Two Regents College faculty members with expertise in Biology nominated faculty 
external to the College for participation. The nomination procedure resulted in the 
selection of three faculty reviewers with no prior affiliation with Regents College. These 
faculty were from public and private institutions in the states of Ohio, Florida, and 
Pennsylvania. 

Materials 

The curriculum for the biology major in each of the ten comparison institutions was 
outlined adjacent to the Regents College biology curriculum (see Table 1 for the Regents 
College curriculum structure), resulting in 10 fating sheets. Reviewers were also asked to 
make a global rating of equivalence of the Regents College biology curriculum as 
compared to the 10 comparison institutions and their own home institution. 
Acknowledging the importance of the curriculum review in context, additional materials 
about Regents College were provided to the external panel. Among these materials were 
a Liberal Arts catalog, a copy of the Annual Report to the Faculty from the Academic 
Vice President, a listing of distance learning courses available to students obtained from 
the College's Distance Learn database (http://www.lifelongleaming.com), and sample 
status reports (i.e., transcripts) of recent biology graduates. Since course titles vary 
greatly across institutions, course descriptions for each of the ten comparison institutions 
were also provided, along with a brief description of each institution. Reviewers were 
also asked to give an overall rating of currency of the Biology concentration curriculum 
structure. For purposes of the review, curricular currency has been defined as the degree 
to which the curriculum under evaluation “compares to the current research and thinking” 
in a particular discipline. Therefore, to be current, courses in the curriculum must 
represent those topics considered seminal and reflective of the changes in the field over 
time, such that new approaches to a topic are reflected in the course opportunities for 
students. 

In addition, through its Outcomes Assessment Framework (Peinovich & Nesler, 
2000), Regents College has developed a set of learning outcomes, called objectives, for 
each of its external degree programs. Reviewers were also asked to evaluate the learning 
objectives for the Biology concentration (see Table 2) in terms of their equivalence and 
currency as compared to their knowledge of the field and their home institutions. 



O 

ERIC 



159 



I G 9 



Procedure 



External faculty were nominated and contacted for their willingness to participate in 
the review. Upon the decision to participate, each panel member received the ratings 
packet and supplemental materials. The actual reviews were conducted individually and 
the panel “met” via two teleconferences. Materials were sent to faculty in advance of the 
first teleconference to allow time for review of the materials prior to the discussion. 

The first teleconference served as an orientation to the college and to the curriculum 
review process. Discussion of the college’s history, mission, characteristics of the 
student body were discussed. In the time between the first and second teleconferences, 
faculty completed their ratings packet, comparing the Regents College biology 
curriculum to the curriculum of the selected peer institutions and making judgments as to 
its equivalence and currency. Overall ratings of the curriculum equivalence and currency 
were also obtained. In addition, program objectives were also rated for their overall 
equivalence and currency. All ratings were made on a seven-point scale ranging from 1 
(not at all equivalent/current) to 7 (very equivalent/current). 

One week after the introductory teleconference, during the second teleconference, a 
discussion of the ratings of each individual peer institution along with overall ratings 
were reported and discussed. Strengths, weaknesses, and recommendations for changes to 
the curriculum followed the ratings discussion. A report was drafted based on the 
recommendations of the external panel for presentation to the faculty for review and 
consideration. 



Results 

During the second teleconference, the external review panel reported difficulty in the 
task of assessing equivalence of the curriculum. Discussion of the ratings indicated that 
lack of equivalence between the Regents College biology curriculum and biology 
curriculum of the peer institution could be a function of rigor in either curricula. Thus, 
the reviewers recommended that the scale anchors be changed to "Not at all Rigorous" 
and "Very Rigorous" for future reviews. As a result, the following discussion of the 
reviewers' comments regarding the curriculum is qualitative in nature. 

In most cases, the reviewers reported that the Regents College Biology curriculum 
was equivalent to the biology curriculum of their home institution. The external panel 
cited the required history of science or bio ethics course as a major strength of Regents 
College Biology curriculum. Other strengths included the requirement for a course in 
developmental biology and the breadth of choice in the curriculum. The molecular 
biology requirement was also noted as being current. Two weaknesses were reported by 
the panel: (1) the absence of a course emphasizing Biodiversity, and (2) the possibility of 
substituting a course in Evolution for a course in Genetics. 



In terms of the program objectives, the external panel indicated that few programs 
outline such objectives for their programs, but that the objectives appeared to be 
reflective of the curriculum structure. This was found to be a strength of the Regents 
program. The panel thus indicated that the objectives were equivalent and current. One 
recommended change to the objectives was to change made the word “systematic” to 
“systems” biology in Objective #4 (see Table 2). 

Based on the discussion, the panel recommended the following curricular changes: 

(1) to make Genetics a required course in intermediate and upper-level courses (Level II), 
removing Evolution as an alternate choice; (2) to move Evolution to the electives level 
(Level III) of the curriculum; (3) to insert a course in Biodiversity into the core level 
(Level I); (4) to change "Systematic Biology" in Level EC to "Systems Biology"; and (5) 
to revise objectives as needed based on the above recommendations. 

Discussion 

The curriculum review process provided constructive feedback about the biology 
concentration curriculum as structured. The external reviewers generally indicated that 
the Regents College Biology concentration was quite comparable to that of the traditional 
four- year comparison institutions and to their home institutions. Recommendations 
included that genetics be a required course without the opportunity use evolution as a 
substitute course. Next, the panel recommended the insertion of a Biodiversity course 
into the core requirements. Finally, a revision of the language of “systematic” biology to 
“systems” biology in the curricular structure and program objectives in biology was 
proposed as a change. The Liberal Arts Faculty voted to approve each of the 
recommendations at their Fall 1999 meeting. 

Overall, the procedure utilized ran smoothly. The first teleconference was initially 
anticipated to last approximately 30-60 minutes. However, the teleconference length was 
approximately two hours. While it was longer than anticipated, the length allowed for the 
development of rapport among reviewers, thus facilitating conversation in the second 
teleconference, which was also about two hours long. External reviewers engaged in 
lively discussion of the curriculum while acknowledging the mission of the College. 

With respect to modifications in the review process itself, the rating scale anchors 
were changed for subsequent reviews. Faculty panel members indicated that the task of 
rating equivalence was difficult because the curricula of the two institutions could be 
nonequivalent but for different reasons (i.e., strengths or weaknesses in either curricula). 
Because the goal of the review was to ensure that students completing an external degree 
program at Regents College were obtaining an equivalently rigorous academic 
experience, the scale anchors were revised to read “Not at all Rigorous” and “Very 
Rigorous.” To facilitate ratings in subsequent reviews, an adapted definition of rigor 
(Spahn, 1998) was adopted by the Liberal Arts faculty, such that rigor has been defined 
as “ strong base of knowledge and understanding through a thorough and challenging 
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learning experience” (Liberal Arts Faculty, 1999). The change in anchors will hopefully 
decrease the ambiguity in the rating task. 

In summary, outcomes of the review were viewed as positive from the perspective of 
the faculty and external panel, and the process itself was economically feasible. The 
curriculum structure review process itself is recommended as one step in the overall 
review of a program which balances economic feasibility with the qualitatively rich 
information provided for program improvement. 



Table 1 

THE REGENTS COLLEGE BIOLOGY CONCENTRATION CURRICULUM 

STRUCTURE 



I. Core Required Courses 

A. Introductory Biology 

B. Cell/Molecular Biology 



n. Required Areas (Choose at least One Course from each of the 
following areas) 

A. Genetics & Evolution 

B. History of Science/ Bioethics 

C. Systematic Biology (Animal/Plant) Including Anatomy & 
Physiology; Intermediate Botany, Vertebrate Physiology; Histology 

D. Ecology 

E. Development (Embryology, Developmental Biology) 



HI. Electives 

Total Credit Hours: 30 Hours (15 of which must be upper level) 
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Table 2 



THE REGENTS COLLEGE BIOLOGY PROGRAM OBJECTIVES 

1. Describe the essential functions of cellular systems and the interrelationships of 
organisms and populations. 

2. Define and apply the underlying principles of genetics or explain current theories of 
evolution. 

3. Demonstrate an understanding of major innovations in the history of science or 
analyze current problems in bioethics using a variety of currently held assumptions. 

4. Demonstrate upper level knowledge of systematic approaches to the study of life 
forms. 

5. Demonstrate knowledge of ecological systems. 

6. Demonstrate knowledge of modes of development among life forms. 
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THE IR-CQI CONNECTION 



Tracy Polinsky 

Coordinator of Institutional Research 
Butler County Community College 



Introduction 



What is CQI? 

Like its industrial counterpart TQM (Total Quality Management), Continuous Quality 
Improvement, or CQI, became popular a few decades ago in the United States. It was 
presented as a means of achieving organizational excellence, and many jumped on the 
bandwagon. As is common with approaches du jour, "quality" appealed to many, was 
embraced by some, and was seriously adopted as a way of conducting business by few. 
Today, CQI is alive and well at certain institutions of higher education, who are making a 
conscious and ongoing effort to integrate CQI philosophies and tools into their problem- 
solving and process-improvement endeavors. 

Many view CQI as a rigid formula to which they must adhere; however, there is 
nothing magical about CQI in and of itself. Quality means excellence. CQI then, involves 
striving for excellence (good enough is not good enough) and continuously trying to 
improve oneself or one's institution. 

In order to better itself, an institution must first identify areas for improvement. The 
primary way to identify these areas is through assessment. Whether it is of a quantitative 
or qualitative nature, this assessment must yield accurate and reliable information on 
which decisions can be based. Because institutional researchers are by nature evaluators 
and collectors of data, they are a logical and valuable part of any college or university's 
CQI team. 

This Institutional Researcher's Experience with CQI 

In December 1998, 1 was asked to join the CQI Steering Committee at Butler County 
Community College. The committee's mission was twofold. Members were to monitor 
the effectiveness of college committees and to serve as official CQI experts and trainers 
for the campus community. The Steering Committee was comprised of 13 individuals, 
several of whom underwent intensive CQI "Trainer's Training" in spring 1999. The 
mission of the emergent Training Team was to help groups solve specific problems using 
the CQI approach. As a result, individuals would not only leave with practical solutions 
to their present problem, but would also be able to apply CQI strategies to other problems 
or processes. Since then, this CQI Training Team, of which I am part, has facilitated four 
problem-solving or process-improvement "workshops" at the college: 
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• A Scheduling Assessment Meeting (July 1999) arranged by the President for the 
President, President's Cabinet, and invited guests. The Training Team led 
participants through two days of examining the college's credit course schedule 
and arriving at ways to increase enrollment by making adjustments to the 
schedule. Since then, several of the ideas have been implemented at the college. 

• A project conducted by the Advising Task Force (started in October 1999, 
ongoing). The Training Team led the task force through an examination of the 
current advising process and obstacles to successful advising. The group is 
currently exploring various advising models to determine which model would 
address these issues and work best at the college. 

• A Service Excellence workshop conducted on the college's Professional Day 
(February 2000) for all front-line staff. Members of the Training Team engaged 
the group in various activities and taught the group CQI principles and strategies 
for continuous improvement. The Training Team helped participants develop 
service themes and standards of excellence for their work areas as well as ways to 
evaluate their success. 

• A project undertaken by the CQI Steering Committee to improve the 
communication process at the college (initiated in February 2000, ongoing). The 
Training Team led the group through an examination of communication at the 
college. Once the root causes of ineffective communication were uncovered, the 
group addressed them and developed a model for effective communication that 
has been recommended to the President for implementation. The Training Team is 
also teaching other committee members to become CQI trainers. 

Why Should an Institutional Researcher be Involved with CQI? 

Scientific Approach 

CQI is deeply rooted in the scientific approach. Whether it is in problem 
identification or problem solving, a systematic approach is imperative. Processes must be 
carefully observed, studied, and documented, and the root causes of problems identified 
(as opposed to symptoms or "obvious" causes). Successful statisticians and researchers 
are by nature conscientious investigators and recorders of data and events. They know the 
importance of documentation and how much it will mean down the road. They are 
methodical and know how to collect valid, reliable, meaningful, and pertinent data. 

CQI can be represented by the PDCA (Plan - Do - Check - Act) Cycle. In the 
planning stages of a CQI project, a problem-solving or process-improvement strategy is 
developed. "Do" refers to the implementation of the plan. During the checking phase, the 
phenomenon is studied to see if the implemented change made a difference. The group 
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then adjusts its strategy during the "Act" phase, which then leads back to "Plan" and so 
on. This cycle can be likened unto a scientific experiment wherein the strategy is the 
independent variable and the phenomenon of interest (ex. enrollment) is the dependent 
variable. Of course, unlike rigorous scientific experiments, it is nearly impossible to 
control for extraneous variables in a real-life college setting. 

Data 

In quality efforts, decisions are no longer based on hunches or anecdotal information, 
but on sound data. Here, the importance of the institutional researcher on a CQI team can 
not be overstated. They are skilled at a) collecting data, b) analyzing data, and c) 
communicating data. 

Often, individuals believe information is needed but do not know how to obtain it. 
The institutional researcher usually knows if the data already exist and the best way to 
procure information when it does not. They know how to design and conduct surveys, 
focus groups, and the like. They also know how to collect data properly, that is, to ensure 
that the data collected is valid and reliable. 

Once the data are obtained, they must be understood. Namely, the data must be 
manipulated so that they are capable of answering the group’s question(s). There are 
responses to be interpreted, data to be entered, and statistics to be applied. Researchers 
are also good pattern spotters and theme identifiers. 

Finally, it is not enough for the data analyst to understand the information; he/she 
must be able to effectively communicate it to others. Institutional researchers are well 
versed in the art of data reporting, having experience presenting data in virtually every 
format -- written and oral reports. Power Point presentations, tables, charts, and flip- 
charts. Most importantly, they know how to present information in a way that is 
understandable, logical, and relevant to their audience. 

Customer Focus 

CQI maintains a customer-oriented philosophy. Because institutions of higher 
education exist to provide services to their customers (students, community, etc.), 
they must be confident that their customers are pleased. In CQI, as in IR, customer 
feedback is an essential component of the improvement process. Institutional 
researchers understand the importance of obtaining feedback (especially when 
calculating response rates). A large part of their jobs entail administering satisfaction 
surveys to the institution's customers, primarily its students and former students. 
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Assessment 



At numerous points during a project, a CQI team relies on assessment. At the 
beginning stages, the current situation must be assessed. In later stages, the CQI team 
must evaluate proposed solutions. But the bulk of evaluation takes place during the 
"Check" phase of the PDCA Cycle. While this assessment does not occur until after a 
plan or solution is implemented, it must be mapped out during the initial planning stage. 

Institutional researchers are valuable if not necessary components of a CQI team if 
for no other reason than to guide the team through assessment planning. While most 
individuals on the CQI team are familiar with assessment in a general sense, they are 
usually not proficient at formulating an effective assessment plan from scratch. 

Measurable objectives or outcomes are a critical component of any assessment plan, 
yet writing such "operational definitions" is not a skill that comes naturally to most 
people. Some may have never been exposed to such a thing, while others are simply out 
of practice. At any rate, the institutional researcher can assist them in this process. 

When planning for assessment, evaluation criteria must be written. These criteria will 
later help the group determine if the implementation of their solution(s) helped them to 
achieved their goal(s). They must know what is to be measured, how to measure it, and 
how they will know if their solution was successful. Because institutional researchers are 
experienced measurers and writers of such objectives, they can not only facilitate the 
group's composition of the objectives, but also teach them these skills directly. 

Mission 

A focus on the mission of the institution is imperative to CQI. Projects and plans 
must be aligned with the institution's purpose, and individuals must be committed to not 
only meeting but also exceeding the college’s goals. The institutional researcher, if 
involved in institutional effectiveness activities, already understands the foundational 
nature of the mission. She/he knows how to derive measurable outcomes from an 
institution's mission and objectives and how to collect data to determine if the institution 
is achieving its goals. 

Resistance 

It is natural for many individuals to resist assessment and to fear change. Many 
perceive assessment as a faultfinding mission, and hence a threat to their security and to 
the status quo. They may approach CQI efforts with caution, trepidation, or outright 
resistance. Institutional researchers face these challenges every day, at times worse than 
others. They realize the importance of introducing assessment and change slowly and 
carefully into an organization's existing system. And hopefully they have acquired a 
sensitivity to the concerns of others and have found ways to successfully assuage them. 
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An Example of the IR-CQI Connection in Practice 

In summer 1999, the college President requisitioned the services of the newly formed 
CQI Training Team. He asked the team to facilitate a study of the credit course schedule 
and its possible effects on enrollment. Invited to attend this "Scheduling Assessment 
Meeting" were members of the President's Cabinet and other guests. 

Before the meeting, the Training Team spent many hours reviewing what they had 
learned and preparing for the project. Being the first time to lead a group through a 
project via CQI, the preparations were arduous and exhausting. In fact, it was at this point 
that one of the six original members of the Training Team resigned. Eventually, the team 
developed a plan that would seemingly address the issues and satisfy its charge. 

Pre-Meeting 

The CQI Training Team assembled relevant student data on the topic. These data 
consisted of the results of student surveys and focus groups. In short, the information 
revealed the most critical issues surrounding class scheduling from the perspective of the 
students. 

Also prior to the Scheduling Assessment Meeting, the Training Team asked 
participants to collect data from others in their divisions. Questions such as "How is the 
credit course schedule developed?" and "What factors influence enrollment?" were used 
to generate discussion. Participants were asked to bring this information with them when 
they attended the meeting. Thus, data collection was the first step in the CQI process, 
allowing the group to analyze the current situation before engaging in process 
improvement. 

Introduction 

The first part of the meeting involved an introduction to the topic and a statement of 
objectives. The Training Team also introduced participants to Continuous Quality 
Improvement, including an orientation to effective teamwork, the improvement cycle 
(Plan - Do - Check - Act), and some basic CQI tools. 

Identifying Relevant Issues 

Brainstorming was used as a means of generating many ideas. Participants were 
asked to record responses to the question, "What issues must be considered when 
developing and implementing the credit course schedule?" Student data and data 
collected from participants' staff were incorporated at this point. 
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The CQI Training Team then led the group through an Affinity Diagram whereby 
ideas generated from brainstorming were clarified, discussed, and clustered according to 
common themes. 

Identifying Root Causes 

Once major themes were identified, root causes of less-than-maximum enrollment 
(with respect to the credit course schedule) were sought. An Interrelationship Digraph 
helped the group to examine and graphically portray the cause and effect links among the 
"idea clusters" generated. Once completed, the group was able to determine which 
scheduling issues were affecting enrollment at the most fundamental level. 

Establishing Evaluation Criteria 

Participants received instruction in evaluation, including clarification and explanation 
of evaluation terminology. Enrollment data were also presented that enabled the group to 
identify benchmarks and goals. The group was then led through the development of 
Evaluation Criteria, via Brainstorming and 10-4 Voting (a CQI decision-making tool). 
During this stage, the group determined how they would measure enrollment during the 
later "Check" phase of the PDCA Cycle. The Evaluation Criteria established would later 
determine to what extent the implemented solutions accomplished the group's 
objective(s). 

Identifying and Choosing Solutions 

Brainstorming and an Affinity Diagram were again used to generate then group all 
possible solutions. Participants then composed solution statements for each of the clusters 
that described the actions that would need to be taken. 

The CQI Training Team next facilitated the establishment of Decision Criteria against 
which the solution statements were judged. These criteria served as a "reality check" for 
the solutions generated, by asking, in a sense, if the recommended actions were "do-able" 
and worth the effort. The participants voted to prioritize the solutions. 

Conclusion 

The importance of evaluation was underscored. The meeting was summarized, and 
the group’s original objectives were revisited. The group discussed what actions would 
be taken after the meeting. Finally, the CQI Training Team asked all participants to 
complete an evaluation of the meeting itself. These results were later analyzed and 
reviewed by the CQI Training Team who used them to improve their own training 
efforts. 
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Post-Meeting 



Decisions were made by the appropriate individuals to implement several of the 
solutions generated at the meeting. No official assessment has been conducted at this 
point. A CQI newsletter was designed and issued in May 2000, which informed the 
campus community of the status of the project. 

My Unique Contribution to the Project as an Institutional Researcher 

• Data collector. As an institutional researcher, I was the resident data "expert." I 
knew what data existed, where to get it, and how to get it. This applied to both 
quantitative and qualitative data. I readily knew what the data meant -- and what it 
didn't mean. And when data was needed that did not exist, I knew how to devise a 
way to get it. 

• Theme identifier. I was able to recognize emerging themes quickly and easily, 
particularly during the Affinity Diagram activities when a plethora of ideas had to 
be clustered and "boiled down." I believe this ability comes primarily from my 
qualitative research experience, but also from my experience as a trend and 
pattern spotter, survey researcher, environmental scanner, and general data 
analyst. 

• Relationship identifier. Whether we are drawn to institutional research because 
we are scientific and analytical, or whether our jobs make us this way... the 
bottom line is that researchers have certain characteristics. We understand 
relationships between and among variables. This skill was an asset particularly 
during the group's search for root causes of the phenomenon. I understood the 
cause and effect relationship between items, and I knew what could and could not 
be concluded based on the given data. 

• Assessment specialist. Nowhere was my existence (as an institutional researcher) 
more critical to the project than in evaluation. Because I spend nearly half of my 
time planning, conducting, interpreting, and reporting the results of assessment, I 
have become one of the college's evaluation "experts." I was able to educate the 
group in evaluation concepts and terminology, as well facilitate their writing of 
evaluation criteria (outcomes). 

CQI Learning Experiences 

Wow, what a trip. When mere babes, we were charged by the President to facilitate a 
project to increase enrollment. No pressure there! I would be lying if I said it was easy. It 
was stressful, demanding, and exhausting, but above all it was time consuming. Although 
subsequent CQI projects have become easier, they have all been time intensive. The 
amount of time spent preparing for CQI workshops is beyond anything we ever imagined. 
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I can tell you that if you are considering joining (or initiating) CQI efforts at your 
institution, you must be prepared to work hard. And only join if you are an intrinsically 
motivated person. 

Now for specifics. We learned to be prepared for anything and to be flexible enough 
to change our course when necessary. Some portions of the workshop may take less or 
more time than expected — be prepared to adjust quickly. You may notice in the 
afternoon that the participants are "brain-dead." We did, and decided to call it quits and 
set up a second session for another day. The CQI Training Team simply used the time in 
between to review what had transpired and plan accordingly for the next meeting. 

In the worst case scenario, a tool you have chosen to use may "flop." In other words, 
it may not accomplish your goal. This happened to us during the Scheduling Assessment 
Meeting. We had chosen to use a Fishbone to uncover root causes. While the Fishbone 
was an excellent tool in theory, it became literally too large to manage. (Since then, we 
have made modifications to it and have used it successfully.) We ended up taking the 
major ideas that resulted from the Fishbone and switching to the Interrelationship 
Digraph to address them. We used this opportunity to show the participants that we were 
monitoring and constantly improving our own training program, which was very CQI-ish. 

As thorough as you are, you will never think of everything. We had planned our 
training without realizing the limited knowledge the participants had regarding 
assessment and goal setting. The vacant stares I received during the Evaluation Criteria 
phase indicated that we needed to educate them before we could proceed. So between the 
first and second sessions, we put together a "lesson" on evaluation and how to write 
measurable outcomes. Once they were taught the necessary information (a refresher for 
many, I suspect), they were ready to dig back in. 

We learned logistical things, like how to best utilize the physical space of the meeting 
room. We have become experts at table, prop, and poster arrangement. We know how big 
the lettering needs to be for people from a certain distance to read it and what kind of 
markers do not bleed through onto the walls. 

Then there were the "little things." We have found that these are precisely the things 
that can make the biggest difference. For example, we always put bowls of goodies on 
the tables at the beginning of the day, filled with gum, mints, chocolate, aspirin... People 
love it! 

Despite the foibles, the CQI Training Team believed that we did a pretty darn good 
job conducting our first session. And the feedback confirmed it! The evaluation forms 
that participants (anonymously) filled out contained some suggestions for improvement. 
But overall, they indicated that the participants thought very highly of the work we had 
done. In fact, many went out of their way to personally thank us for our efforts, which we 
greatly appreciated. 
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Conclusion 



Given the amount of time and energy required to produce successful CQI problem- 
solving and process-improvement sessions, why would anyone voluntarily put 
himself/herself through this? The answer is simple. We believe that the CQI approach 
results in better decision making. It is not a panacea. Personally, I see it as a system that 
forces individuals to solve problems and improve processes in a logical and systematic 
way. In other words, I see it as a way of thinking rather than as a set of techniques. The 
tools are there primarily to promote sound thinking. 

As with anything, in order for CQI endeavors to be successful, the right players must 
be assembled. I have found that people with certain personality traits and values are well- 
suited and appreciated members of a CQI team. In addition, particular individuals are 
valuable if not essential to a college's CQI efforts because of their unique experiences. 

The presence of an institutional researcher (or similar person) will make a difference. 
Their knowledge, skills, and understanding in scientific and systematic problem solving, 
data collection and analysis, and evaluation will undoubtedly aid an institution that is 
committed to Continuous Quality Improvement. 
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Abstract 

In response to student and faculty complaints about the amount of time available to 
travel between classes, an analysis of the time between classes problem was conducted at 
a large, public research university. Using facilities, course scheduling and student survey 
data, we discovered that many students had distances to travel between classes that would 
normally take longer than the allotted ten minutes. This forced them to leave class early, 
arrive to class late or skip class altogether and often left them with an inadequate amount 
of time to complete exams. These analyses supported a decision to implement a policy 
regarding student scheduling. 



Introduction 

Colleges and universities across the country are increasingly focusing their attention 
on the classroom behavior of students. A recent article in the Chronicle of Higher 
Education (Schneider 1998) suggests a rise in uncivil behavior of college students that 
ranges from arriving late to classes to physical assaults on faculty. One faculty member 
believes that the current generation of college students is more apathetic than in the past 
and is more likely to display uncivil behavior than ever before (Sacks 1996). Other 
research indicates that classroom incivilities and disruptions continue to have a 
tremendous impact on classroom learning (Boice 1996). 

The costs of classroom incivilities are high. Not only does the increasing frequency 
of uncivil behavior impede the learning process, it also causes students to grow more 
“uninvolved, oppositional and combative” (Boice 1996, p. 480). Colleges and 
universities across the country are forming task forces and committees to examine the 
problem of classroom incivility and possible solutions. 

One of the most common forms of uncivil behavior is students arriving late to class 
and leaving early (Boice 1996). Most would agree that these disruptions can be 
attributed to individual student motivation and disinterest (Wyatt 1992); however, on a 
large campus students may be arriving to class late and leaving early because of the 
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distances they must travel to get from one class to another. Does the common ten-minute 
interval between classes give students enough time to get from one side of the campus to 
another? While a great deal of attention has been paid to students’ reasons for disrupting 
class, little research has been done to assess the impact of distance between classes and 
classroom disruptions. 

If the allotted time between classes were not enough, many campuses would be faced 
with a difficult and perhaps costly policy decision. Colleges could simply choose to 
accept students’ tardiness and change scheduling practices by increasing the amount of 
time between classes. To make such a significant change in scheduling would create 
logistical challenges and cost perhaps thousands of dollars to implement. Colleges could 
also take measures that would attempt to change student behavior. Either option is 
certain to be difficult and costly. Before making such a dramatic policy decision, 
colleges would be wise to assess the impact that distance between classes has on students. 

The University of Maryland, College Park, a large, public research university, was 
faced with such a policy decision. Students had become increasingly vocal about the 
difficulties they experienced arriving to class on time when they had only ten minutes to 
walk across campus. Given that the campus is approximately two square miles and 
consists of more than 400 buildings, few faculty and administrators were skeptical of the 
problems students were encountering. In addition, faculty were complaining of 
disruptions in class due to students arriving late and leaving early, and some faculty 
claimed that students had approached them with concerns about arriving to class on time 
when they were faced with only ten minutes to make large treks across campus. 

A campus committee of administrators and faculty was appointed by the Provost to 
address the issue of distance between classes. The committee was tasked with 
understanding the extent of the time between classes problem and its impact on campus. 
Understanding the extent of the problem was especially important given the substantial 
costs of proposed changes to the class schedule. The campus had not performed any 
previous analyses on this topic, so we set out to collect and analyze data that would 
inform and assist the committee in their decision making. 

Approach 

In collecting reliable information for the task force, we combined “hard” data from 
the university course scheduling system with “soft” student survey data. To understand 
the extent of the problem we first estimated the time it takes to walk between classes 
using Fall 1999 facilities and course data. We then used the data to classify 
undergraduate students into three groups: students with no Monday-Wednesday-Friday 
(MWF) back-to-back classes, students with MWF back-to-back classes who could travel 
between the classrooms in ten minutes or less, and students with MWF back-to-back 
classes whose travel time between the classrooms was greater than ten minutes. 
(Tuesday-Thursday classes were not considered because of their longer fifteen minute 
break between classes.) 
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These three groups of students were surveyed via email and the Internet to determine 
their support for changing the course schedule as well as their actions in response to the 
time between classes problem, and why they chose a course schedule that made it 
difficult to travel between classrooms. The survey was conducted at the beginning of the 
Spring 2000 semester and comprised an initial email describing the location of the survey 
website, followed by three followup emails. The response rate was 40%. 

Calculating times and distances between classes 

To understand how long it takes to travel between classrooms across the campus, the 
Office of Records & Registration initially approached the problem by having students 
actually time how long it took to walk between pairs of buildings. The magnitude of this 
effort quickly became apparent and the project was abandoned. There are almost 4,500 
unique building pairs for courses taught during the Fall 1999 semester, and having 
someone walk and time the distances between all the pairs was simply not practical. 
Indeed, this was the major stumbling block for the project, and we were forced to develop 
an alternative method to measure the times and distances between classrooms. 

Our solution was simple. We combined the two dozen building pairs that had been 
measured by Records & Registration with an estimated distance for each pair to run a 
bivariate regression model predicting travel time using estimated distance. We then 
applied the results to the estimated distances for all building pairs to derive an estimated 
travel time for each building pair. This approach allowed us to calculate a reasonable 
accurate travel time that only required measuring travel times between a few building 
pairs. 

At our request, a detailed map was generated by Facility Drawings with a layout of 
100 yards per grid square (see Figure 1). Each grid line on the map was numbered 
starting with zero. The grid coordinates for each classroom building were then 
determined and used to calculate the Euclidean distance (i.e., distance “as the crow flies”) 
in hundreds of yards between each possible classroom building pair. 

From their previous attempt. Records & Registration had already timed 
approximately two dozen trips between building pairs. Combining these times with the 
respective calculated distances in a simple bivariate regression provided an estimated 
walking time per hundred yards of distance. The bivariate regression equation fit the data 
well (R 2 =.88), and according to the model results it takes on average a little over one 
minute to walk 100 yards across campus, a plausible result. 

Using the estimated distances from the grid map calculations and the relationship 
between walking time and distance from the regression model allowed us to estimate a 
travel time for all Instructional building pairs on campus. Two minutes were added to 
these times to account for miscellaneous actions such as bathroom breaks between 
classes, the time it takes to get from building entrances to classrooms, etc. 
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Table 1 shows the distribution of walking times for undergraduate students with 
MWF back-to-back classes during the Fall 1999 semester. The first column gives the 
number and frequency of student/classes per week. For example, if a student has a class 
on MWF that is followed by a class that meets only on Monday, she is counted once in 
this column. If she had a class on MWF followed by another MWF class, this student is 
counted three times. The second column gives the number and frequency of only 
students. Out of the 8,924 students with back-to- back classes, 2,570 (28.8%) have one or 
more back-to-back classes with walking times of 10 minutes or more. These students 
comprise 10.4% of all undergraduates registered during the Fall 1999 semester. 

From the preceding analysis, we can see that the time between classes problem is 
substantial. During the Fall 1999 semester over 2,500 students registered for classes that 
were too far apart to travel between during the ten-minute break. While it is possible that 
these students were still able to travel between classes in the allotted time, such a large 
number of students indicate that class disruptions due to these schedules could be 
significant. 

Survey data 

Student responses to not having enough time to travel between classes are listed in 
Table 2. Students in the third group, those students registering for at least one pair of 
MWF back-to-back classes in classrooms greater than a ten-minute walk apart, were 
asked their actions in response to their back-to-back class schedule. Students were 
allowed to choose more than one action. Only 23% responded that they had enough time 
traveling between classes. The most common student response was leaving class early, 
with over half the group indicating that they chose this course of action. About 12% 
indicated they arrived for class late, and about 11% simply skipped class. Disturbingly, 
almost 40% stated they had difficulty completing examinations because of their 
schedules. 

The survey results indicate that over three-fourths of the students estimated to have 
problems traveling between classes did indeed have problems. Most of these students 
reacted by leaving class early, with smaller proportions arriving to class late or skipping 
class altogether. 

In addition, we note that these disruptions are not randomly distributed amongst all 
class types. Because juniors and seniors will be taking a larger proportion of courses that 
satisfy their major, and because courses within a major tend to be taught within the same 
one or two buildings, freshmen should be more likely to have problematic class 
schedules. Using our estimated data from Table 1 and student class, of the students with 
back-to-back classes on MWF in Fall 1999, freshmen were less likely to have ten minute 
or less travel times than upperclassmen. About 65% of freshmen with back-to-back 
classes had travel times of less than ten minutes, compared with 71% of sophomores, 

76% of juniors and 76% of seniors. 
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Finally, we asked students why they constructed class schedules that made it difficult 
to travel between classrooms in the allotted ten minutes. Students were again allowed to 
choose more than one reason. The results are presented in Table 3. The two most 
common reasons were related to the courses themselves: either one of the courses was a 
required course or it was the only course offered at the time needed. Interestingly, the 
third most popular response was “wanted a compact course schedule.” Many students are 
registering for back-to-back courses not only out of necessity, but also out of 
convenience. From the focus group where we pilot tested the student survey, many 
participants said that students schedule these back-to-back classes so that they won't have 
any “wasted” time by having a half-hour or hour between classes. 

Conclusion 

These data provided a great deal of insight into the problem of the time between 
classes and informed the task force committee about several aspects of the problem. 

First, the data indicate that the amount of time between classes is a significant problem at 
the University of Maryland, College Park. According to our analysis, a large proportion 
(over one fourth) of the students taking back-to-back classes on MWF do not have 
enough time between classes to arrive on time to their next class. 

Second, the limited time students have to travel from one class to another is affecting 
the learning process. It causes students to leave class early, arrive to class late, and skip 
classes altogether, impacting both the individual student and the classroom as a whole. 
The time between classes also appears to limit the contact some students have with 
faculty. Most alarming is that a large proportion of students indicate they have 
encountered difficulties in finishing exams due to the time they have to travel between 
classes. Freshmen are also more likely to have back-to-back classes in rooms far apart. 
Given the importance that the first year of college has on the success of students, this is 
of particular concern. 

Students indicated that they selected their back-to-back scheduling for many reasons, 
but most did so out of necessity. Students were forced to schedule classes due to limited 
offerings, major course requirements, and time conflicts. However, students’ reasons for 
scheduling back-to-back classes also indicate that they do so not only because of the 
unavailability of courses but also out of convenience. Many students want compact 
schedules and appear to recognize the problems in scheduling back-to-back courses. 
Given that today’s students often work to help pay for their education, it is not surprising 
they would want a compact schedule that allows them to pursue those efforts. As the 
traditional college education where students reside on campus and attend school full-time 
gives way to students living at home and working while attending college, administrators 
must increasingly take into account the external pressures faced by students when 
determining scheduling policies. 

Armed with our analysis, the committee was faced with a difficult policy decision. 
They had two options: they could either accept student scheduling behavior and change 
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the current scheduling system, or they could keep the current schedule and take measures 
that would attempt to change student behavior. With so many students experiencing 
problems, the committee knew that some action must be taken to help alleviate the 
problem. 

Given the costs and the enormous task of changing scheduling practices by increasing 
the time between classes from ten to fifteen minutes, the committee recognized the 
impracticality of allowing more time between classes. They recognized that a problem 
did exist and began to search for other solutions. One inexpensive solution was to use the 
results from our distance-time analysis to flag students at registration who may have 
problems. Currently, the University is working to implement a warning into the 
registration program that will notify students when they have scheduled back-to-back 
classes that are more than a ten-minute walk apart. So, as student register for classes by 
phone or the Internet, they will be warned when they are scheduling back-to-back classes 
that may be in buildings that are too far apart. 
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Figure 1. University of Maryland, College Park Facilities Map with 100-Yard Grid 
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Table 1. Distribution of the Time Between MWF Back-to-Back Undergraduate 

Classes, Fall 1999 



Time between classes 


Student/classes per week 


Students 


Number 


Percent 


Number 


Percent 


Less than 10 minutes 


13,251 


69.9% 


6,354 


71.2% 


10- 10:59 


1,555 


8.2% 


659 


7.4% 


11 - 11:59 


1,168 


6.2% 


540 


6.1% 


12-12:59 


623 


3.3% 


336 


3.8% 


13-13:59 


862 


4.5% 


408 


4.6% 


14-14:59 


819 


4.3% 


326 


3.7% 


15-15:59 


408 


2.2% 


184 


2.1% 


16 minutes or more 


264 


1.4% 


117 


1.3% 


TOTAL 


18,950 


100.0% 


8,924 


100.0% 
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Table 2. Student Reactions to the Time Between Classes Problem 



I left class early. 56 . 6 % 

I arrived at class late. 12.1% 

I skipped class because I was running late. 10.7% 

I had difficulty completing in-class examinations. 39.0% 

I was unable to speak with the instructor after class. 1 1 .0% 

I did not have any problems getting to class on time. 23. 1 % 



Note: N=290. Question: “Which of the following did you tend to do 

because of this back-to-back class schedule? 
Please check all that apply.” 
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Table 3. Reasons Why Students Schedule Back-to-Back Classes in Rooms Far Apart 



Accommodate my work schedule. 25.2% 

Accommodate family schedule. 3.1% 

At least one is a required course. 49.7% 

Only course offered at the time I needed. 43. 1 % 

Only course available when I scheduled classes. 30.3% 

Wanted a compact schedule. 37.6% 

Limited course offerings. 24. 1 % 

Had other scheduling conflicts. 35.2% 

Transportation issues (bus, metro, car pool, rush hours, etc.). 6.2% 

Other. 5.2% 



Note: N=290. Question: “Why did you schedule these two courses back-to-back? 

Choose as many reasons as apply.” 
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ASSESSING THE ASSESSMENT DECADE: WHY A GAP BETWEEN 
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The North Central Association of Colleges and Schools has stated concisely that 
“Programs to assess student learning should emerge from, and be sustained by, a faculty 
and administrative commitment to excellent teaching and learning” (NCA, 2000, p. 32). 
But excellence seems to represent a moving target. As Winona State University 
Assessment Director Susan Hatfield (1996) points out, the validation of excellence in 
higher education has shifted from an earlier emphasis on inputs and processes to a more 
recent focus on outcomes. This fundamental change, I believe, is only one of several 
imbalances in the practice of assessment that cry out for equilibrium. 

A highly-respected treatise on assessment concludes with an epilogue entitled, “A 
Matter of Choices.” The authors write that assessment can be conducted in various 
legitimate ways. “As such, the process of planning and implementing assessment 
programs requires many choices,” between philosophically different alternatives. 
However, these pairs of alternatives need not be seen as mutually exclusive; in fact, they 
should complement each other in striking “a balance that works” (Palomba and Banta, 
1999, p. 331). They discuss three critical sets of choices that institutions must face in 
quest of balanced assessment: 

• improvement versus accountability as motivations for assessment; 

• quantitative versus qualitative means of assessment; 

• course-based versus non-course models of assessment. 



Half of my time is spent as a Professor of Political Science at West Liberty State 
College, where I have served for three years as Co-Chair of our College Assessment 
Committee, exposing me to many of the asymmetries found in assessment practices. My 
instincts as an instructor tell me that Palomba and Banta are right when they support 
equilibrium, or homeostasis, as desirable concerning the choices cited above. I would go 
even further, and suggest that when gross imbalances exist, they belie something akin to 
pathology in academe. 

When I look around at current practices at my home institution, at the other 
institutions in West Virginia, and nationally (as recounted in books, a major research 
survey, and journals like Change, Research in Higher Education and Assessment 
Update ), I see a system rife with disequilibrium concerning these three vital issues. That 
is, a system motivated more by accountability than desired improvement, employing 
quantitative techniques far in excess of qualitative ones, and conceptualized chiefly as 
non-course-based. What troubles me about the status quo is that it symbolizes a jarring 
disconnect between: (1) the inclusive theory of assessment; and (2) the equally exclusive 
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practice of assessment. The American Association for Higher Education’s first principle 
of good practice says that “the Assessment of student learning begins with educational 
values” (AAHE, 1989, p.2), and my educational values tell me that these imbalances are 
unhealthy 

The assessment movement practically owned the decade of the nineties in higher 
education. However, the “assessment of assessment” undertaken in a recent survey of 
1 ,393 institutions, conducted by the National Center for Postsecondary Improvement, or 
NCPI, chronicles decidedly unimpressive results (Peterson and Augustine, 1999). As the 
first major study asking exactly what institutions do with the extensive data that previous 
studies say are being gathered on our campuses, the NCPI authors want to know if 
assessment data is used profitably, because the assessment literature itself posits that 
student assessment should not become an end in itself, but rather, serve as a means to 
improve education. The NCPI’s baseline conclusion is that “student assessment has only 
a marginal influence on academic decision-making” (Peterson and Augustine, 1999, p. 
21). Among the many valid questions raised by this research are descriptive and 
prescriptive ones about the nature of the faculty role in gathering and using assessment 
data. 

Key Institutional Researchers trumpet the axiom that assessment works best when 
faculty-driven, and Palomba and Banta underscore the point when they posit that “faculty 
members’ voices are absolutely essential in framing the questions and areas of inquiry 
that are at the heart of assessment” (1999, p. 10); but current practice almost seems to 
mock this proposition. Another prestigious group of authors asserts that “it is fact that 
most faculty still have not considered the assessment of student outcomes seriously” 
(Banta, Lund, Black, & Oblander, 1996, p. xvii). The 1999 NCPI study (Peterson and 
Augustine, 1999) concurs, reporting that only 24 percent of institutions say faculty 
members involved in governance are very supportive of assessment activities. An earlier 
Middle States Association survey (MSA, 1996) found that fear of the unknown, plus 
heavy workloads, contribute to pervasive faculty resistance to assessment. I agree that 
unreflective inertia on the part of some professors represents a genuine problem for 
assessment, but faculty reluctance to change explains only part of the problem. Even if 
every instructor in America reads Spencer Johnson’s (1998) best-selling parable 
depicting humanity’s penchant for fearing the unknown, then meditates on its insights 
(change happens, anticipate change, monitor change, adapt to change quickly, change, 
enjoy change, be ready to change again), widespread faculty support for assessment will 
not suddenly materialize. 

Many professors actively engaged in assessment have expressed thoughtful criticisms 
regarding the current modus operandi. In particular, instructors lack confidence in 
assessment’s relevance (applicability to classroom teaching and learning), validity (truly 
measuring learning outcomes), proportionality (institutional benefits of assessment 
commensurate with effort devoted to it), and significance (answering the question that 
comes naturally to academics: So what?) Addressing these issues is essential for the 
movement’s goal of an assessment culture developing on-campus. Based on my own 
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experience I would hypothesize that many faculty, though involved in assessment, have 
failed to prioritize it above competing agendas. And what results from relegating 
assessment to such second-class citizenship? Deferring initiative for assessment to 
administratively-oriented professionals who typically are not teachers. 

For those professors truly infected by the virus of skepticism, one antidote consists of 
a large dose of qualitative methods, or soft data. Assessment’s practitioners have clung 
to quantification like David Letterman to velcro, a syndrome critics call the data lust 
fallacy. The 1999 NCPI national survey found that the norm consists of institutions using 
“easily quantifiable indicators of student progress and making only limited use of 
innovative qualitative methods” (Marchese, 1999, p. 54). Yet, it strikes me as naive for 
institutional researchers to expect over-reliance on empiricism to capture the hearts and 
minds of dubious instructors. 

One pair of advocates for greater reliance on qualitative assessment argues that a 
pervasive myth needs to be disputed. This myth assumes that, since qualitative methods 
communicate in words rather than numbers, they are not as rigorous. The authors 
contend, however, that “These methods, when applied with precision, take more time, 
greater resources, and certainly as much analytical ability as quantitative measures” 
(Upcraft and Schuh, 1996, p. 52). Another observer notes that the flexibility of qualitative 
techniques allows them to operate in a more natural setting and “permit the evaluator to 
study selected issues in depth and detail” (Patton, 1990, pp. 12-13). A sub-text reason why 
assessment has featured quantification may be that numbers are more easily processed by 
state legislators and external govemors-those powerful individuals vigorously applying 
pressure for institutional accountability. 

Once the cod-liver-oil of soft data helps to balance the campus assessment cocktail, 
my second antidote for the virus of skepticism infecting some faculty is an equally 
healthy dose of course-related process and content. Put simply: process relates to the 
heuristic “how” of teaching and learning; content refers to the heuristic “what” of 
teaching and learning. These issues embrace what faculty know and care about, and they 
are also expressed in language congenial to the professoriate. The standard approach of 
using standardized tests to measure student outcomes in areas such as math, writing 
skills, critical thinking, and computer literacy is useful, but insufficient. Free-standing 
outcomes testing entails a feedback loop back to the classroom that is too amorphous. 

Practitioners relying on outcomes testing exclusively exhibit something of the myopia 
lampooned by Plato in his “Allegory of the Cave.” Plato’s mythic prisoner, chained in a 
manner allowing him to see only the shadows of life on the cave wall-not life itself— 
parallels those willing to settle for the shadows of the educational process, as opposed to 
education itself. The 1999 NCPI research supports this line of reasoning, finding that 
“relatively few links exist” between measures of student assessment and the faculty’s 
classroom responsibilities. Germane to this gap is Palomba and Banta’s assertion that 
“integrating assessment activities into the classroom and drawing on what faculty are 
already doing increases faculty involvement” (1999, p. 65). Emulating best-practices 
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rather than worst-practices is essential, and an NCA Assessment Consultant recently 
praised Winona State University for the clever incentives devised there to foster faculty 
participation in assessment activities (Lopez, 2000). Not coincidentally, the half-time 
director of assessment at Winona State, Susan Hatfield, spends the other half of her time 
teaching in the Communications Department. 

Therefore, pedagogical process and content pertinent to the faculty mind-set ought to 
be blended liberally into the assessment mix. But too seldom does this happen. A well- 
known advocate of Classroom Assessment Techniques (CAT) contends that the one- 
minute paper (now used in over 400 courses at Harvard) provides valuable feedback from 
student to instructor, quickly and efficiently, making it an example of CAT worth 
emulating (Cross, 1998). One program steeped in CAT operates at Raymond Walters 
College (University of Cincinnati), and uses the course grading process for both 
departmental and general education assessment. Notably, the mind behind assessment at 
Raymond Walters is a chemistry professor, Janice Denton, who splits her time between 
the classroom and administering assessment. Her consultancy at my home institution 
impressed me as replete with creative ideas. However, a meaningful spillover effect at 
this institution eludes detection. My sense is that the key players (Department Chairs) 
accept many of Denton’s ideas, but don’t know how to apply the concepts to their own 
bailiwick. I believe that the rigorous course syllabus can provide concrete hooks to 
grounds assessment in the classroom experience that Department Chairs understand and 
value, thus I have begun conducting seminars there on the relationship between model 
syllabi and assessment. 

The other half of my time is spent at West Virginia University, serving as Co- 
Director of a statewide international studies consortium (FACDIS), which includes all 20 
of West Virginia’s public and private institutions. This role has given me an appreciation 
for the potency of improved course syllabi to enhance both faculty and course 
development. For two decades, FACDIS has relied on improving course syllabi as its 
principal means of holding faculty accountable. The consortium involves 375 faculty 
from more than 15 disciplines in projects funded by a combination of state funds and $1.5 
million from competitive external grants. FACDIS has received two prestigious national 
awards in the process. 

The vital resource of an exemplary course syllabus can link assessment to the 
classroom, and it can also generate innovative soft data germane to pedagogical process 
and content. A recent article develops the case for more sophisticated course syllabi 
(Strada, 2000). Just as the last thing a fish would notice is water, academics tend to 
overlook the value of a comprehensive course syllabus. It seems too prosaic for some 
higher education professionals to take seriously. But despite operating largely in 
obscurity, a nascent body of literature appreciative of the syllabus’ diverse contributions 
is beginning to emerge (Altman and Cashin, 1992; Birdsall, 1989; Grunert, 1997). The 
only book-length treatment of syllabi considers course content, course structure, mutual 
obligations, and procedural information as basic necessities, but advocates a truly 
“reflective exercise” serious enough to improve courses by clarifying hidden beliefs and 



assumptions as part of a well-developed philosophical rationale for the course (Grunert, 
1997). Ideally, I look for part of a professor’s academic soul to shine through the pages of 
a thoughtful syllabus. 

The potential benefits of creating more complex syllabi fall into three categories. First 
and foremost,- good syllabi enable student learning by improving the way courses are 
taught. This benefit seems transparent to veteran instructors who have worked to improve 
a syllabus and know how it adds efficiency to organizing the course, saves time in future 
semesters, and establishes a paper trail to highlight the good things they already do in the 
classroom. Such intuitive insights are bolstered by a study examining commonalities 
found among Carnegie Professors of the Year recognized by the Council for 
Advancement and Support of Education (CASE). University of Georgia Management 
Professor John Lough spawned the idea of dissecting the behavior of CASE Professors of 
the Year to see what makes them tick— a form of best-practices benchmarking. The 
universal common denominator cited by Lough is that “Their syllabi are written with 
rather detailed precision. Clearly stated course objectives and requirements are a 
hallmark. They employ a precise, day-by-day schedule showing specific reading 
assignments as well all other significant requirements and due dates” (Lough, in Roth, 

Ed., 1996, p. 196). 

Closely related to energizing teaching and learning is a second benefit of 
sophisticated syllabi that remains more opaque to academic eyes: use in faculty 
evaluation. A recent book purporting to explain every aspect of exercising the duties of a 
Department Chair, fails to include the word syllabus in its index, nor could I locate the 
word syllabus in the book’s 279 pages (Learning, 1998). An elegant syllabus includes 
lesson plans that provide the only true road map of what is really being taught, and, how 
it is being taught, in that course. The concept of a lesson plan is dismissed too summarily 
by higher education faculty and administrators as pertinent only to secondary schools 
(therefore beneath us). 

Yet, my experience tells me that lesson plans help to establish an upward course 
trajectory from semester to semester because the process is a cumulative one: you no 
longer backslide by forgetting something effective that you did five years ago, or, by 
failing to ground a trial balloon that didn’t fly last time out. In the one course that I teach 
every semester, I revise lesson plans immediately after class. In this way, a lesson plan 
evolves in ways analogous to the process of pecking away at a script. Precise lesson plans 
also represent something of a pedagogical insurance policy for institutions with aging 
faculty. For example, at my home institution, a majority of professors in the School of 
Liberal Arts are older than 55. If illness strikes, good lesson plans would help to protect 
the academic integrity of what transpires in the professor’s absence. Because the 
comprehensive syllabus and its lesson plans are under-appreciated, it is not surprising 
that academic administrators rarely grasp the syllabus’ pertinence to promotion and 
tenure decisions. 
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Completely absent from the assessment script is any hint that the exemplary course 
syllabus is a player on the academic stage. This is unfortunate, because a fine syllabus 
contains what is tantamount to the DNA code for an endangered species: qualitative 
assessment that is creative and relevant to curricula. Curricular structures matter, and the 
solid planning endemic to worthy syllabi yields dividends that can help to bolster 
curricular integrity. Even more importantly, dense syllabi allow us to forge substantive 
links between the three curricular levels of the academy which researcher Robert 
Diamond says currently proceed in random directions: individual courses, programs of 
study at the departmental level, and general education programs at the institutional level. 
The disconcerting result, claims Diamond, is that most free-wheeling curricula “do not 
produce the results that we intend”(1998, p. 2). Another higher education analyst 
similarly bemoans the curricular randomness noted above, suggesting that “institutions 
tend to frame policies at the global level, leaving the specifics of learning to disciplines 
comprised of single courses, and those disciplines seldom have the necessary resources” 
(Donald, 1997, p. 169). 

Linking these curricular levels in meaningful ways can only occur by holding faculty 
accountable, but doing so without violating their academic freedom-which is sure to 
happen once you tell them what they should teach (content), or, how they should teach it 
(process). Only sophisticated syllabi provide detailed and accurate snapshots of how 
content and process come to life in the classroom. Only thoughtful syllabi afford 
instructors the breathing space to reveal their pedagogical essence, thus facilitating 
scrutiny, but without rigid or heavy-handed directives. Only serious syllabi provide 
extensive soft data to augment the hard data typically generated to satisfy demands for 
curricular accountability emanating from oversight bodies. I am passionate about the 
virtues of solid syllabi because I have seen them bear fruit: in the efforts of the FACDIS 
consortium, and in my own classroom. However, while sophisticated course syllabi can 
be used for either faculty evaluation or assessment purposes, it is a cardinal assessment 
principle that these two processes should function separately at any given institution, to 
avoid the possibility of conflict of interest between assessment and faculty evaluation. 

Assessment professionals can facilitate the course syllabus emerging as the fulcrum 
linking the three levels of the academy. In order to do so, they would benefit from 
insights gleaned from educational psychologist Robert Sternberg (1995). He attacks 
standardized testing (typically used in higher education assessment) for its failure to 
incorporate the vital element of creativity. Thirty-one years as a teaching professor in 
higher education have convinced me that the value of creativity in solving academia’s 
problems remains ill-appreciated. 

Academics seem to have big left-brains, but small right-brains; the academy loves 
science, but mistrusts experiential insight. Consequently, higher education tends to 
undervalue creativity. A counterpoint to this tendency materialized recently when the 
President of my home institution, Ronald Zaccari, received the American Association of 
University Administrators’ Eileen Tosney Award, given annually for “administrative 
innovation.” The AAUA noted Zaccari’ s work in art, especially sculpture, as contributing 
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to his innovative efforts. In 2000, he presented a keynote address to the Association of 
Institutional Researchers, challenging IR people to think more creatively. In seconding 
this motion, I recommend balancing assessment with more soft data, concern for 
improvement of instruction, and the creation of course-based efforts. Fortunately, the 
sophisticated course syllabus can be employed to realize each of these worthy ends more 
comprehensively than the portfolios and capstone courses usually cited in the literature as 
examples of creative assessment. 

In conclusion, the Institutional Research literature’s best-case scenario-that 
assessment efforts be faculty-driven-makes good abstract sense. However, in the real 
world of widespread faculty skepticism about assessment, wisdom counsels that IR 
professionals nurture faculty support more creatively; preferably where they live-in and 
around the classroom. The common polemical cement housing both administrators and 
faculty is still damp enough to preclude predicting the future with any certainty. Four 
plausible scenarios still seem capable of materializing during the next decade: 1 ) 
assessment as faculty-driven; 2) assessment as faculty-supported; 3) assessment as 
faculty-tolerated; 4) assessment as faculty-denigrated. In my view, the first option 
represents as ideal type that will occur rarely under special circumstances. The second 
option is certainly feasible, if assessment practitioners make an effort to engage the issues 
of relevance, validity, proportionality, and significance that rankle the professoriate. I see 
the third option as a reflection of the status quo, and likely to continue unless more 
creative thinking is exhibited by all concerned. However, the worst-case scenario of the 
fourth option should not be discounted as impossible. Realistic faculty know that the age 
of accountability will not soon disappear, but unless assessment is constructively linked 
to the courses they teach, even their acquiescence cannot be taken for granted. 

The North Central Association’s extensive, decade-long review of assessment in 1999 
concludes somberly (much like the NCPI) that “In institutions where key faculty have not 
claimed ownership, or participated wholeheartedly and in large numbers, they have had 
great difficulty in launching and developing their assessment programs” (Lopez, 1999, p. 
9). The report places a great deal of emphasis on the potency of opposition by “faculty 
leaders” (as opposed to rank-and-file faculty) in this comprehensive NCA document. This 
corrosive problem of influential senior faculty speaking out against assessment is 
something that “institutions are reluctant to bring up in conversation or written 
documents,” but if not carefully defused, can become the “most persistent and 
deleterious” of all the obstacles to successful assessment (1999, p. 11). 

From my perspective, it looks like the assessment literature is unaware of another 
valuable resource directly relevant to this issue. If administrators are hypothetically from 
Mars, then many faculty in higher education are from Venus. Hailing symbolically from 
different planets, the chasm between these denizens of academia can be bridged 
creatively by those relatively few split personalities, like Janice Denton (Raymond 
Walters College) and Susan Hatfield (Winona State University), who hold academic rank 
and teach half-time while running exemplary assessment programs as their alter-ego to 
the classroom. As a practitioner of this 50/50 model of time-structuring for the past 21 



years, I have labeled this dichotomy as the Lokai role (named for the character played by 
Larry Storch in the original Star Trek series). Lokai is black on his left side and white on 
his right side, exactly opposite of a rival race colored white on the left side and black on 
the right side. To Star Trek’s audience, of course, Lokai and his bitter enemy seem barely 
distinguishable— but to the protagonists-they might as well come from different planets. I 
understand that some risks exist for people like Janice Denton and Susan Hatfield who 
play a Lokai role on-campus. However, these are personal political risks. For those 
serendipitous institutions having individuals performing Lokai roles, the chances of 
making assessment operate in ways congenial to faculty values are better if they exploit 
this resource than if they do not. 
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Introduction 

This paper examines the structural/organizational characteristics associated with 
positive student performance, learning, and growth at two and four-year institutions. 

The importance of this research is based on three external forces. First, accrediting 
agencies (Middle States Association of Colleges and Schools, Western Association for 
Schools and Colleges’ Commission for Senior Colleges and Universities, North Central 
Association of Colleges and Schools) have been revamping their policies to stress student 
learning (McMurtrie, 2000). The Council of Regional Accrediting Commissions recently 
drafted new standards of accreditation that include a “...focus on student learning instead 
of institutional preferences” (p. A58) (Camevale, 2000). A review of the guidelines and 
mission statements of accrediting agencies reveals the inclusion of student outcomes as 
an important component of the accreditation process. This research augments these 
current initiatives by identifying some of the structural/organizational characteristics 
related to student learning. A second growing force for higher education is the 
emergence of performance indicators in state funding (Cabrera & La Nasa, 2000). 

This research demonstrates structural/organizational characteristics that address 
student performance, learning, and growth. Structural/organizational characteristics are 
measured by the size of the institution, revenues, expenditures, endowment, selectivity, 
complexity, and the presence of residence students. These measures are aligned with the 
current literature on organizational effectiveness (Hall, 1991; Lewis, 1995; Pascarella and 
Terenzini, 1991; Reiss, 1970; Volkwein, Valle, Blose, & Zhou, 2000). 

These characteristics can help state government and other funding sources identify 
potential performance indicators that will enhance student performance, learning, and 
growth. Although structural/organizational characteristics such size, wealth, complexity, 
and selectivity may not initially appear to be indicators, many of the indicators currently 
included in state performance budgeting criteria (for example, SAT scores, array of 

204 




197 



academic programs and services, revenue enhancement strategies, and targeted 
populations) contribute to these four categories of characteristics (Burke, 1997). Finally, 
Cohen and Brawer (1996) identified a major gap in research between four-year 
institutions and two-year institutions. Compared to studies of four-year institutions, there 
is a relative dearth of research on the two-year sector. The proposed research will 
address this gap, and the articulate some of the commonalities and differences between 
these institutional types. 

This research addresses the following questions: 

1 . Controlling for other variables what are the structural/organizational 
characteristics of institutions that contribute to positive student performance, 
learning, and growth? 

2. What are the differences between two-year and four-year institutions that most 
contribute to positive student performance, learning and growth for a 
population of fourth semester students? 

Conceptual Framework 

The Pascarella (1985) model of student outcomes provides the conceptual framework 
for this study. The Pascarella (1985) General Causal Model specifies five elements 
influencing student learning and cognitive development. These elements are 
structural/organizational characteristics of institutions, (size, mission, wealth, complexity, 
and selectivity), student background/pre-college traits (aptitude, personality, ethnicity, 
high school experiences), interactions with agents of socialization (faculty and peer 
interactions), institutional environment (classroom experiences, student services, 
tolerance, safety), and quality of student effort. The Pascarella model assumes that all 
these components contribute directly or indirectly to learning and cognitive development. 

The study examines on the structural/organizational characteristics of institutions 
controlling for factors such as the institutional environment (as perceived by the student), 
the interaction with agents of socialization, the quality of student effort, and pre-college 
traits. This allows the authors to examine the influence of specific 
structural/organizational characteristics associated with student learning and growth, as 
well as the differentiation or similarity between the two-year and four-year institutions. 

Method 



Participants 

This research utilized a 1997 multi-campus database drawn from 51 public (23 four year 
and 28 two year) institutions. There are 7,658 students in the database who completed 
the assessment instrument at the end of their second year. The study is limited to second 
year students ensuring that students have spent an equal amount of time at their 
respective institutions. 
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Pascarella’s (1985) General Causal Model 






Materials 



The database contains both institutional and student level data. The institutional level 
data includes information on the organizational complexity, financial resources, 
selectivity, sector, residential component, and student demographics. The student level 
data includes information on pre-college characteristics, perceptions of the institutional 
environment, experiences of academic and social integration, financial aid, effort, and 
student learning and growth. 

Institutional measures of wealth, enrollment, and sector were obtained from the 
Integrated Postsecondary Educational Data System (IPEDS). Institutional complexity 
measures were gathered from the Directory of Higher Education (1997 edition). 

Procedure 

The institutional data was gathered from multiple sources, all for the 1997-1998 
academic year. A committee of cooperating researchers and administrators from 
participating institutions developed the survey instrument. The instmment is grounded in 
the Pascarella (1985) and Cabrera, Nora, and Castaneda (1993) models of student 
outcomes and persistence. 

The Cabrera, Nora, and Castaneda (1993) model of student persistence proposes a 
more complex array of factors than the Pascarella (1985) model resulting in student 
persistence decisions. Included in the Cabrera, Nora, and Castaneda (1993) model are 
financial aid, pre-college academic performance, significant others encouragement, 
financial attitudes, academic and intellectual development, grade point average, social 
integration, institutional commitment, goal commitment, and the intent to persist. This 
model provided additional factors to be included in the database that have been 
demonstrated to have significant relationships with student persistence. 

The survey for the database was printed and scored by the American College Testing 
program. The database was analyzed using SPSS pc version statistical software. 

Measurement 

There are a number of variables and constructs hypothesized to be related to student 
performance, learning, and growth contained in the database. The present study 
examines the variables and constmcts proposed by both the Pascarella and Cabrera 
models of student outcomes related to student performance, learning, and growth. 
Specifically, the variables included in the study include the following variables, also 
listed in Table l 24 . 
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Table 1 referenced in this paper may be obtained by contacting the authors. 



Dependent Variables 

Learning and Cognitive Development 

For the purposes of this study, student performance, learning, and growth is taken 
from two perspectives: students and faculty. First, student perceptions of growth are 
obtained from students’ self-assessment of their own intellectual growth (acquiring 
information, ideas, concepts, and analytical thinking) on a five point growth scale (1= 
none and 5= extremely high). Second, faculty perceptions of student learning were 
measured by the cumulative grade point average reported by students. 

Independent Variables 

Structural/Organizational Characteristics of Institutions 

Key indicators for structural/organizational characteristics used in previous literature 
have included size, wealth, complexity, mission, and selectivity (Volkwein, Valle, 
Parmley, Blose, & Zhou (2000). Size is represented by the total undergraduate headcount 
enrollment at the institution. Mission is measured on a scale from 1 to 6, with 1 being 
Associate degree granting, and 6 being Professional degree granting. Wealth includes 
measures of revenues and expenditures per annual full time enrollment. The complexity 
measure reflects the number of organizational units headed by a Vice President or Dean 
(or equivalent) and the highest degree offered by the institution. Selectivity includes the 
percentage of applicants admitted. 

In addition to these factors, this study included the presence or absence of residential 
housing on campus. 

Pre-college Factors 

This study controls for student characteristics such as racial/ethnic group 
membership, disability, gender, previous employment, dependent children, 
socioeconomic background, age, SAT score, high school rank, and high school average. 

Interactions with Agents of Socialization 

This study also includes student reported variables reflecting the extent of interactions 
including the amount of faculty interactions (amount of direct contact with faculty, 
satisfaction with faculty and advisors) and the extent to which the students interacted 
with their peers (extent and value placed upon peer interactions). 

Institutional Environment 

Factors contributing to institutional climate include measures of classroom 
experiences (stimulation in class, faculty quality, classroom satisfaction), perceptions of 
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openness and tolerance (satisfaction with the atmosphere of understanding, freedom from 
harassment, racial harmony, understanding of lesbian/gay/bisexual issues, and 
security/safety), perceptions of low prejudice (by peer students, faculty, and 
administrators), satisfaction with various student services, and satisfaction with various 
academic support services and facilities. 

Quality of Student Effort 

Student effort is measured by student perception of good study habits and giving a 
high priority to studying. 



Data Analysis 

First, a factor analysis was conducted to see if the items clustered consistently with 
student outcome theory. The resulting factors were examined. Resulting scale 
construction is reported in table 1, and scale reliabilities for the two-year sector, the four- 
year sector, and the combined sample are reported in Table 2. 



Table 2. 






2 AND 4 




2 YEAR 


4 YEAR 


YEAR 


FACULTY 

INTERACTION 


0.81 


0.74 


0.79 


PEER INTERACTION 


0.85 


0.87 


0.86 


INVOLVEMENT 


0.76 


0.74 


0.75 


LOW PREJUDICE 


0.92 


0.88 


0.91 


OPEN TOLERANCE 


0.76 


0.71 


0.73 


HEALTH SERVICE 


0.87 


0.83 


0.79 


REGISTRATION 
AND BILLING 


0.79 


0.68 


0.74 


CLASSROOM 

EXPERIENCE 


0.89 


0.88 


0.89 


STUDENT EFFORT 


0.80 


0.79 


0.79 


GROWTH 


0.88 


0.87 


0.88 



The principle method of analysis is the use of OLS regression equations to predict the 
dependent variables, grade point average and student growth. Separate regression 
equations for the dependent variables are run for the two-year, four-year, and combined 
populations. 



Results 

Controlling for other variables what are the structural/organizational 
characteristics of institutions that contribute to positive student performance, 
learning, and growth? Tables 2 and 3 display the regression beta weights for each of 



the three populations with GPA as the dependent variable in table 2 and student self- 
reported growth as the dependent variable in table 3. In each case, the variables were 
entered in blocks consistent with the Pascarella model (pre-college variables first, 
structural/organizational variables second, interactions with agents of socialization third, 
institutional environment and effort fourth). 

The results indicate that the structural/organizational characteristics of mission, 
complexity, and residential percentages do contribute to student performance, learning 
and growth. Specifically, for both the four-year sector and the combined populations, the 
higher the degree offered by the institution, the lower the gpa of its students. The more 
wealth an institution has, the higher the GPA for students at four-year institutions, and the 
more complex a four-year institution is, the higher the students’ performance and 
learning. Finally, the more students live off campus in the combined sample, the better 
their grade point average. In terms of growth, the higher the degrees offered by the 
institution, the more growth the students reported experiencing. 

What are the differences between two-year and four-year institutions that most 
contribute to positive student performance, learning and growth for a population of 
fourth semester students? Difference do exist in this sample between the 
structural/organizational characteristics contributing to positive performance and learning 
between the two and four-year sectors. While none of the structural/organizational 
characteristics were related to student performance and growth at two-year institutions, 
four-year institutions demonstrated significant relationships between their mission, 
wealth, and complexity and performance and learning. Specifically, students at four-year 
institutions that offered lower degrees experienced greater learning and performance. 
Additionally, four-year institutions with greater wealth had students with higher reported 
learning and performance. Finally, students at four-year institutions with greater 
complexity reported higher learning and performance than students at four-year 
institutions with less complexity. 
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Table 3. 



DEPENDENT VARIABLE: COLLEGE G.P.A. 



2 YEAR 4 YEAR 2 & 4 YEAR 







N=5082 


N=2576 


N=7658 






Beta 


Beta 


Beta 


PRE-COLLEGE VARIABLES 


RACIAUETHNIC GROUP 


0.079* 








TOTAL SAT 


0.230*** 


0.275*** 


0.250*** 




HSRANK 


0317 *** 


0.258*** 


0.288*** 




TOTAL R-SQUARED 


0.264 


0.171 


0.183 




STRUCTURAL/ORGANIZATIONAL 


MISSION 




'-0.345*** 


'-0.266*** 




WEALTH 




0.111** 






COMPLEX 




0.244* 


0.139* 




COLLEGE RESIDENCE 




' 


0.100** 




TOTAL R-SQUARED 


0.277 


0.215 


0.226 




AGENTS OF SOCIALIZATION 


PEER 




-0.096** 


'-0.057* 




TOTAL R-SQUARED 


0.306 


0.255 


0.257 




INSTITUTIONAL ENVIRONMENT 


CLASSROOM 




0.123** 


0.102** 




EFFORT 


0.193*** 


0.237*** 


0.220*** 




TOTAL R-SQUARED 


0.333 


0.300 


0.294 




*p<.05 


**p<0i 


***p<.001 
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Table 4. 



DEPENDENT VARIABLE: STUDENT GROWTH 







2 YEAR 


4 YEAR 


2 & 4 YEAR 






N=5082 


N=2576 


N=7658 






Beta 


Beta 


Beta 


PRE-COLLEGE VARIABLES 


GENDER 




0.080*** 


0.071*** 




AGE 






-0.045* 




TOTAL SAT 


-0.083* 




-0.065* 




STUDENT WITH 
DISABILITY 




‘-0.085** 


-0.046* 




TOTAL R-SQUARED 


0.062 


0.063 


0.080 



STRUCTURAL/ORGANIZATIONAL 


MISSION 






0.140** 




TOTAL R-SQUARED 


0.067 


0.092 


0.099 



AGENTS OF SOCIALIZATION 


PEER 


0.192*** 


0.254*** 


0.237*** 




TOTAL R-SQUARED 


0.509 


0.461 


0.490 



INSTITUTIONAL ENVIRONMENT 


INVOLVM 


0.169*** 


0.109*** 


0.134*** 




REGBILL 


0.080* 




0.048* 




CLASSROOM 


0.440*** 


0.399*** 


0.406*** 




EFFORT 






0.048* 




TOTAL R-SQUARED 


0.512 


0.462 


0.492 




*p<.05 


**p<.01 


***p<.001 





Summary, Conclusions, and Significance 

The results of this study demonstrate that structural/organizational differences do 
influence student’s performance, learning, and growth. While it is extremely importance 
to keep focusing on the academic preparation, interactions with agents of socialization, 
institutional environment, and student effort to influence student performance, learning 
and growth, accreditation agencies, state governments, and institutions themselves should 
pay attention to the issues of mission, complexity, residence component, and wealth. 

Equally important is the demonstration that two and four-year sector institutions are 
not the same when it comes to predicting student performance, learning, and growth, and 
hence should not governed, evaluated, or monitored according the same standards. The 
results of this study indicate that accreditation, funding, and governing bodies should 
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examine two-year and four-year institutions separately, and create separate criteria for the 
assessment of the two sectors. 

Specifically, for the two-year sector, student pre-college characteristics predict 
approximately one-half of the R2 variance for grade point average. This indicates that 
what the students bring to the two-year institutional environment has tremendous 
implications for their subsequent performance and evaluation. In contrast, the four-year 
sector had a greater variety of influences on grade point average. Pre-college 
characteristics, institutional type, student effort, and classroom environment all 
contributed substantially to grade point average. 

In reference to student growth, the profiles between the two sectors were more similar 
than the profiles between the two sectors for grade point average. Classroom 
environment appears to be much more important in predicting growth than any other 
variable. This finding supports recent research (Volkwein, Valle, Blose, & Zhou, 2000) 
that the classroom experience is a critical variable in student outcomes. 

This research can contribute to the current discourse regarding the transition of 
accrediting agencies to a more student learning centered perspective. The significant 
results indicate that some institutional factors can contribute to student performance, 
learning, and growth, potentially influencing the criteria used in accreditation processes. 

Second, the study contributes to the continuing issue of performance indicators in 
higher education. The key institutional factors associated with increased effectiveness of 
student performance, learning, and growth, could serve as performance indicators for use 
by state governments for funding initiatives. Because much research conducted on 
student outcomes fails to examine the two-year sector, or compare the two vs. the four- 
year sectors, much of the rich information is overlooked. This information is critical 
when creating performance indicators. The difference between the regression outcomes 
for the two and four year sectors in this study indicates that when performance indicators 
are established, the different institutional sectors should be taken into consideration. 

Third, the study provides a critical comparison of student outcomes in the two and 
four year sectors. Little research has directly compared the effectiveness of two and four 
year institutions and the factors that comprise such effectiveness. The present study 
identifies those factors for each sector and compares them, demonstrating that differences 
between the two sectors do indeed exist. 

Limitations of the Study 

Generalizability of the results of the study may be limited due to single state, public 
institutions participating in the study. Additionally, the results are limited to the 
population of second year students included in the study for analysis. These second year 
students represent only those students who have successfully persisted at their respective 
institutions. Results from this study may not be generalizable to students who do not 



persist through their second year. This persistence may also be related to institutional 
type (i.e. two vs. four-year institutions). 

Although using grade point average has become accepted as a measure of student 
learning, it may not be the best indicator possible (Pascarella, 1985). Hence, the results 
are limited to the belief that grade point average is an adequate proxy for student 
learning. 

Third, the database does not include items related to the degree of sophistication of 
institutional technology, a structural/organizational characteristic that may be related to 
student learning and growth. 
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USING QUALITATIVE ANALYTICAL METHODS FOR INSTITUTIONAL 

RESEARCH 

Carol Trosset 

Director of Institutional Research 
Grinnell College 



Introduction 

My early research training was in the natural sciences, primarily observational field 
biology and animal behavior. Then while I was a student at Carleton, I began studying 
cultural anthropology, and I became an ethnographer. There aren’t many ethnographers 
working in institutional research, which seems odd, since what ethnographers do is study 
communities. I do more anthropology as an institutional researcher than I did in my seven 
years as a faculty member, so today I’ll try to show you what that contributes to the sort 
of institutional research that I do. % 

What ethnographers do, specifically, is spend years in an initially unfamiliar 
community gathering masses of apparently unrelated information, most of which is 
qualitative (such as how people behave at public gatherings, or what they say in casual 
conversations on the street). Over time, you try to piece all these things together to build 
an insightful picture of how that society works, how the people in it think, and what 
things they value. Obviously, qualitative analytical skills are central to this effort. 

Qualitative Research as a Process 

This sort of research is an inductive process. That is, you don’t set out to test a theory. 
Instead, you start with masses of information, and theories and answers emerge from it. 
The goal is usually to build what is sometimes called a “grounded theory,” which simply 
means that it emerged from the evidence rather than by being derived from a pre-existing 
theory. While anthropologists often use pre-existing theories to make sense of their 
surroundings, it’s very important to work inductively as well, to guard against becoming 
too enamored of a particular conceptual approach. 

Since social data are usually very complex, one good rule is start with complex data 
and initially look for patterns in the absence of a theory. This is important because you 
usually can’t be sure (especially in an unfamiliar culture) which variables are going to be 
related to the thing you think you’re interested in. Many anthropologists have stories 
about setting out to study one thing, only to be told by the local people that this meant 
they needed to understand something that seemed unrelated. One of my professors went 
to New Guinea to study emotion, and was forced by his hosts to learn all about birds, 
which did turn out to be central to the issue at hand. So one thing ethnographers learn is 
that you can’t know ahead of time what factors are related to each other. 



O 

ERIC 



209 



2,16 



Another key thing about ethnographic work is that you’re often trying to find out 
things that people can’t articulate consciously. This means that you can’t find out what 
you want to know by asking direct questions. Some of the technique lies in knowing what 
indirect questions to ask, or what situations to observe, but the rest of it is hidden in the 
analysis. 

This may all sound very subjective, but there are describable techniques, and there are 
good ways of testing the plausibility of the results. Here’s an example from my first 
research task at Grinnell. 

I was initially hired by Grinnell’ s then-president as a consultant to study why Grinnell 
students felt they couldn’t talk about diversity issues. I and my student assistants did a lot 
of interviews. We compiled a list of issues the students thought it was hard to talk about, 
such as “whether race is an important difference between people.” For each issue, 
students were asked whether or not they wanted to have a balanced discussion about that 
subject. Then they were asked why or why not. At first, I thought this was a failed 
interview design, because so many people misunderstood the question and said no, they 
didn’t want to talk (“have a balanced discussion”) because a discussion of that issue 
wouldn’t be balanced. This meant that we couldn’t get a count of how many people 
thought they wanted to have a balanced discussion, because they hadn’t answered the 
question. But later I went back and assembled all the answers people had given to the 
“why or why not” part of the question. I think I was just being thorough because a few of 
the responses had looked kind of interesting. But when I assembled them all (well over 
100 comments), I found I had overwhelming evidence answering a question I hadn’t 
thought to ask, and which I couldn’t have asked directly anyway: What did students think 
discussion was for ? 

Here are a few representative comments from these interviews: 

• “I want to discuss the importance of sex differences, because I have strong opinions.” 

• “I might discuss discomfort with homosexuality depending on the company. If they 
were persuadable, I would want to convince them.” 

• “I want to discuss causes of sexual orientation because I have strong views on this 
issue.” 

• “I want to discuss religion because I have a unique perspective I like to express.” 

• “I want to discuss the place of religion in society because I have a strong opinion.” 

• “I am not likely to want to discuss the importance of sex differences, but occasionally 
someone needs to be argued with.” 

• “I want to discuss affirmative action because I want to educate people.” 

• “Ideally, you should talk in order to make the other person realize that what they said 
was wrong.” 

• “You should talk in order to reform others to your views.” 

Though each person talked about different issues and used different words to explain 
themselves, there was an amazingly consistent underlying theme. When students wanted 
to discuss something, it was because they held strong views and wanted to convince 
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others. When they didn’t want to discuss, it was because they didn’t know much about an 
issue or didn’t have an opinion. Clearly, discussion was seen as a form of advocacy. I 
went on to identify different dimensions of this assumption. 

One variant takes the form of “The answer is obvious.” 

• “I don’t want to discuss race because it’s not an important difference between 
people.” 

• “I am closed-minded on the importance of race — race shouldn’t distinguish between 
people.” 

• “I don’t want to discuss sexual orientation because it doesn’t really matter.” 

• “I don’t want to discuss causes of sexual orientation because this topic is irrelevant to 
the nature of homosexuality.” 

• “Biological sex has little relevance, there are no major differences, so I would like to 
hear other views (on their importance).” 

• “I want to discuss affirmative action because I want to educate people.” 

• “Affirmative action is a yes or no issue, which makes it difficult for discussion to be 
fair and balanced.” 

Another version goes “I don’t want to talk about things I’m unsure of.” 

• “I would want to discuss multicultural education and affirmative action if I were more 
knowledgeable.” 

• “I’m not sure what multiculturalism is; I don’t know much about it, so I don’t want to 
discuss it.” 

• “I don’t want to talk about multicultural education, because I don’t know what it 
means or what the point is, and therefore I feel uneducated.” 

• “I want to discuss politics as long as I know what I’m talking about.” 

• “I would like to discuss politics if I am knowledgeable about the topic.” 

• “I don’t want to discuss politics because I don’t have a stand on these issues.” 

• “I like discussing gender issues because I feel knowledgeable about them.” 

• “I don’t want to discuss affirmative action because I am not familiar with the 
subject.” 

• “I don’t want to discuss affirmative action because I know absolutely nothing about 
it.” 

• “I don’t want to talk about things I’m unsure of.” 

I also found five whole comments, out of about 200, that assumed a different view of 
discussion, as a form of exploration. 

• “I want to talk about multicultural education because I’m not sure I know enough 
about it.” 

• “I want to discuss multicultural education, as I would like more experience on what 
this would involve. I believe in a broad range of experience.” 

• “I want to discuss race, as it would open my mind to things I don’t experience 
myself.” 



• “I want to discuss multicultural education because I’m curious to see where I stand in 
relation to others.” 

• “I want to discuss multicultural education because it interests me.” 

This is a good example of the inductive nature of this type of analysis. I didn’t even 
know what question I was going to answer by assembling the data. I had never thought 
about what Grinnell students thought discussion was for. Once I saw what they thought, it 
jumped out at me because it was different from what I thought discussion was for 
(namely, exploration; presumably a more common view among intellectuals). Now, this 
is why ethnographers should come from another culture, so that they will notice things 
that are locally obvious. Obviously, at Grinnell I’m not from another culture, but I don’t 
share all the local assumptions, and that’s frequently helpful to me in noticing things 
others take for granted. 

How do I know that I’m right about the students’ view of discussion? There are three 
things that contribute to my certainty. 

First is the fact that I was totally surprised by my own findings. I’d never considered 
the issue and I was astonished that they could think such a thing, so I know, at least, that 
there was no bias internal to myself “trying” to find out what I did. 

Second was the reaction of the other faculty when I reported my findings. They all had 
a sort of “aha” experience. What I said resonated with their own local knowledge. They 
said things like “ that’s what’s been going on in my classroom!” They hadn’t been able to 
articulate it for themselves, but once I did that for them, my conclusions seemed 
immediately obvious and explained much of their own experience. Though the absence 
of this reaction is not always proof that an ethnographer is wrong, when you get a strong 
“aha” reaction it’s always a good sign. And when I presented my findings at conferences 
I got similar reactions from professors at other colleges. 

Third, I did follow-up research using both interviews and surveys, and these studies 
consistently confirmed my initial theories. The kind of analysis I most enjoy doing is 
often most useful in the earlier stages of studying a complex issue. 

When and How to Gather Qualitative Data 

Since most institutional researchers never get the chance to spend as much time on 
one project as I spent on my study of student discussion, let’s look at when to use these 
methods on smaller-scale projects. Most of the time, I do this kind of thing when working 
with either interview data or survey comments. Personally, I vastly prefer interviews to 
surveys, because you can exercise so much more control over whether the person really 
answers the question you meant to ask. Interviews are especially useful under certain 
conditions, including: 

• When you don’t quite know what you want to know 

• When you’re investigating something complex and aren’t sure what questions to ask 





• When you’re trying to study people’s assumptions which they may not be able to 

articulate 

• When you want to do a survey and are trying to make sure you ask the most useful 

questions 

Now that Grinnell has gotten used to having an interviewer in the IR office, I get 
interview assignments on a fairly regular basis. They’re time-consuming, but can yield a 
great deal more information than a survey. 

In one such project, I was asked to interview the most recent three years of tenure- 
track faculty hires, to find out how they had made the decision to accept Grinnell’ s job 
offer. It had occurred to the dean that we always knew someone’s reason for turning 
down an offer, because they told us on the phone when they called to decline, but we 
never heard the reasons why people accepted us. I think that year several of our first 
choices had turned us down, so he sent me out to learn why people accept. Instead of just 
asking them that one question, I got each one to tell me the story of their job search, 
about their other interviews, and what they saw as the pros and cons of coming to 
Grinnell. From this I built up a profile of how many had turned down other offers, how 
many had only applied to Grinnell, how many had already taught elsewhere, and what 
were seen as the most common draws and drawbacks. The analysis was pretty simple, 
since all I had to do was count how many people mentioned each thing, but doing it 
through interviews built up a pretty detailed picture of how people went through the 
process and what issues they were still struggling with. 

Another study required me to interview all the senior humanities majors who had 
taken fewer than three science courses during their time at Grinnell. We have no 
distribution requirements, but most of our students would meet fairly basic ones if we had 
them. So I was sent to talk to the ones who didn’t and find out why not. One of our deans 
had a theory that the only ones who didn’t take science had a good reason, either a 
learning disability or that they took science courses elsewhere in the summers. I was able 
to show that this was not at all true. I documented all the misconceptions these students 
held about the nature of the sciences, and found some places where the advising system 
was not working as expected. I also was able to confirm the widespread faculty 
impression that some students do select Grinnell because they know we won’t make them 
ever take another math course. 

An aside about focus groups, since they are another popular way of gathering 
qualitative data. In the right hands, they can be very effective. I personally don’t use them 
much, partly because they’re far more complicated than they look. They may seem like 
an efficient way to interview a bunch of people at once, and most of the analytical 
process is very similar, but there are enormous complications. This is because the 
members of a focus group are reacting to each other, and it takes great skill to separate 
this out from the rest of what you find. Here are some times when I think interviews are 
better than focus groups: 

• When you have a sensitive topic and people might be reluctant to speak in front of 

others 
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• When you don’t want people to influence each other’s responses (make sure all 
responses are independent of each other) 

• When you’re not sure how to group people in a way that permits effective discussion 

• When you personally are better at paying close attention to one person at a time 

Surveys are, of course, the most common way to gather qualitative data. Most people 
know to leave “white space” on surveys to invite comments, but most people also have 
little experience in how to analyze the things people write in those spaces. Many times, 
the comments get typed up in a list and included as an appendix to a quantitative report. 
This is certainly better than nothing, but there’s much more that can be done. 

Our student affairs office now frequently sends me comments to analyze when they do 
a big survey. Sometimes I’m able to tell them things they didn’t know to ask for. Here’s 
an example. 

Grinnell has a system of Residence Life Coordinators, young adults with master’s 
degrees who live in the dorms. The policy is that they do not enforce rules and are not 
required to report illegal behavior. This system was invented so that students would be 
willing to ask their help when drug or alcohol problems arose. On a recent survey, 
students were asked whether they approved of the policy whereby RLCs did not enforce 
rules. I could have simply added up the yesses and nos and reported that 90% of the 
students approved of the current policy. Since that would have been very boring, and 
some of the comments looked intriguing, I started analyzing the reasons people gave for 
not wanting RLCs to enforce rules. 

I ended up writing a whole report on the student concept of self-governance. 
Technically, student self-governance at Grinnell means that the residents of each 
dormitory floor make and enforce their own rules. This process is supposed to build 
student responsibility. However, I had already been told by students in my own classes 
that anarchy, not democracy, is the dominant model for self-governance. “Self’ is seen as 
referring to each individual, not to a community of students. This information was 
confirmed, and greatly clarified, by the survey comments. As you can see, anarchy, not 
confidentiality, is the dominant rationale. 

Excerpt from Residence Life Survey Report: 

72% of the reasons given contain the (often implicit) argument that no one should enforce 
rules. 

• We are responsible adults. (30%) Translation = “we get to make our own decisions, 
and no one should tell us what to do.” 

• It would violate self-governance to have a non-student, or anyone at all, enforcing 
rules. (16%) 

• The absence of rule-enforcers is good practice for life after college. (7%) 

• There aren’t many problems, so policing isn’t needed. (7%) 

• It feels more comfortable not to have anyone around who can punish you. (6%) 

• The absence of rule-enforcers is an essential feature of Grinnell. (6%) 



I was hoping that this report would demonstrate to the student affairs staff that the 
dominant view of self-governance does little to build student responsibility. As you 
would expect, I failed to cause a revolution, but I did get a few people thinking, at least 
for a little while. 



Content Analysis as a Technique 

Okay, let’s get technical. What do I do with a batch of comments? One reason that 
qualitative analysis gets so much less respect that statistics is that most people don’t 
know that there are any formal techniques for doing it. Courses on how to do it are almost 
non-existent. I never took one. There is a formal technique known as “content analysis,” 
and there have been things written about it. People don’t do it exactly the same way, but 
the approach is consistent enough to describe it. Personally, I learned how to formulate 
the questions and got some practice trying to answer them in several of my undergraduate 
anthropology courses. Then I got lots of practice while doing field research and working 
on my dissertation. Finally, I refined my techniques once I had to do lots of these 
analyses rapidly as an institutional researcher. 

To illustrate how I do this, I’m going to use one of the most conceptually difficult 
content analyses I’ve ever performed. Two years ago, the faculty asked my office to 
study a trial course evaluation form to see how valid and reliable it was. After deciding 
that they wanted to collect comments as well as ratings, they remembered that they had a 
qualitative analyst in the office and asked me to analyze the comments. Here’s what I did. 

The point of analyzing the comments was to use them to test the validity of the 
students’ ratings of the instructors. That is, were high and low ratings confirmed by 
positive and negative comments? I wasn’t sure how to go about doing this, so first I just 
read lots and lots of comments, focusing on the question about the instructor. I found that 
it would be difficult to code whole comments as either positive or negative, since many 
were mixed. I also saw that, although the question asked specifically about whether the 
instructor had contributed to the student’s learning, many of the comments focused on 
other things. So I decided that I would have to classify the comments based on what 
aspects of the instructor the comments were about. 

Now I read the forms again, this time making a list of each kind of comment I found. 

A small sample of this list includes the following: 

Inspiring 

Dedicated to students 

Kind 

Available 

Clear explanations 

Welcomes comments 

Broadened my understanding 

Experienced 



Organized 

Encouraging 

I kept reading until it had been quite a while since I’d found anything that wasn’t 
already on my list. Then I stopped reading and tried to simplify the list by combining 
similar comments into categories. If this were a workshop, we would not break up into 
groups and try this, and then argue about the merits of our various solutions. 

Since this is a paper instead, I’ll walk you through my own work. In the first step, I try 
to put synonyms together. For example, there’s no need to have both “brilliant” and 
“intelligent” on the list. Likewise, “dedicated to students,” “concerned with students,” 
and “respects students” can all be considered the same comment. 

In the second step, I take batches of synonyms and try to link them based on what 
aspect of the instructor they seem to be referring to. For example, 

• Kind / personable / understanding / approachable / warm, 

• Dedicated to students / concerned with students / respects students, 

• Encouraging / helpful / supportive, and 

• Patronizing / condescending / aggressive / threatening 

are all personal qualities referring to the emotional dimensions of interacting with 
students. Notice that positive and negative versions of an attribute belong in the same 
conceptual category, at least at this stage. 

Eventually, I got it down to a list of about ten items. At that point, I read the course 
evaluations a third time, and tried to “code” each professor’s student comments, so that I 
could look for differences between courses. Now, I told you this was a difficult one. It 
took me about three tries to invent a consistent way of scoring each course. In the 
meantime, I had to revise my list of categories a couple of times, because some of the 
original ones turned out not to be mutually exclusive. (You can guess I spent most of the 
summer on this project.) 

At last, I invented a reliable scoring system. In the interests of eventually finishing, I 
asked my colleague (who was doing the statistical analyses of the ratings) to draw me a 
stratified random sample of courses. I coded all those courses, and found that, regardless 
of the question being focused on student learning, most of the comments were about 
other things. 

• 32% were about personal attributes (nice, energetic, available) 

• 30% were about whether the professor was helpful 

• 26% were about perceived competence (knowledgeable, liked how the class was run) 

• 12% were about student learning (made-student think, improved student’s skills) 

Here you can also see the categories that eventually emerged as the things our students 
think about when they evaluate teaching. Most individuals don’t think about all nine 
things, but in a typical class most or all will be mentioned at least once. 

• Professor availability 
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• Professor niceness or approachability 

• Professor energy or enthusiasm 

• Appearance of professor knowledge level 

• How well class sessions were run 

• Whether the student liked the chosen classroom format 

• Whether the professor helped the student understand the course material 

• Whether the course made the student think 

• Whether the student’s skills increased 

Now, the really neat part comes when you combine a qualitative analysis with a 
quantitative one, because then you can see what’s really going on. My original mandate 
was to find out if the comments and the ratings corresponded in any meaningful way. So 
I asked my colleague to do a cluster analysis of the numeric ratings for the courses in the 
sample. He found four clusters, some with different average scores. 

If you stopped there, you’d probably conclude that the professors with an average 
class rating of 4.5 are worse teachers than those in the groups with averages or 5.9 and 
5.6. But look what happens when you combine the clusters with the content analysis. 
Having coded each course for which categories the students commented on, I was able to 
ask whether the pattern of comments coincided with the pattern of scores. To my surprise 
and delight, it did. 

If we were only dealing with the Typical Good Class (well run, helpful professor, 
good course materials, average score 5.6) and the Mixed Feelings/Ambivalent Student 
classes (every student says some things were good and others bad, average score 5.1), it 
might be reasonable to accept the ratings as a good measure of quality. However, the two 
outlier groups have more distinctive features, which are less convincingly linked to 
student learning. 

The high scores (average 5.9) go to the Charismatic Professors. These are the only 
individuals who get many personal comments, and they’re also the only ones who get 
credit for picking good readings. (Others get things like “most of my learning came from 
the readings, not from the professor.”) Students rave about the other students, and about 
the personal relevance of the course material. 

Finally, it turns out that the lowest scores (average 4.5) don’t go to the classes where 
everyone thought there were problems, but to classes with a bimodal distribution of 
comments. (Unfortunately, these did not correlate well with the standard deviation of the 
scores, because that is so small in every class. However, that underscores the usefulness 
of the qualitative analysis.) In these classes, some students raved and others were harshly 
critical of everything. Using my insider knowledge of professors and the curriculum, it 
appeared that many of these courses had content or requirements that in some way 
violated many students’ expectations (like using computers in an anthropology class). I 
also knew that, in these two groups at least, the students’ estimates often did not coincide 
with the respect accorded the instructors by their peers. 





Although some professors found this information very disturbing, others have tended 
to ignore my findings and argue that since the numbers are all right (no statistically 
significant gender bias, etc.), that the forms are a valid measure of teaching quality. I 
continue to argue that validity is about whether the respondents really answer the 
question, and my data show that many of them don’t. It’s been an uphill battle, but I think 
I have at least gotten more people worried about what the ratings really mean, and 
therefore more cautious about how they want to use them. 

In conclusion, qualitative analysis and ethnographic methods generally, and content 
analysis techniques in particular, definitely have something to contribute to institutional 
research. They can be used to illuminate institutional culture, make sense of survey 
comments, and discover things no one has thought to investigate. 
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Purpose 

This paper describes the development and implementation of a student outcomes 
assessment program in a School of Business Administration (SBA) at a public 
baccalaureate institution. Specifically, we will discuss the development of a Primary 
Trait Analysis (PTA) instrument and its implementation within the SBA. The measures 
that were established and monitored through this process will provide valuable 
feedback for improving both school and institutional performance. Several unique 
elements of this assessment effort are that it was designed during the Fall 1999 
semester and implemented in Spring 2000 — a very aggressive timeframe; that it was 
accomplished with the active support and participation of all SBA faculty and 
administration; and that it was implemented at no additional cost to the School or the 
institution. Although the results are still being fully evaluated, the process itself has 
provided useful information, based on student and faculty feedback. 

Background 

The School of Business Administration is one of four Schools within this state- 
assisted baccalaureate level institution of approximately 2,600 students. There are 550 
majors in the School of Business Administration and seventeen full-time faculty 
members. Each major must complete 48 hours in the required business core and 
another 30 hours in a business specialization. The core includes instruction in 
management, marketing, accounting, economics, communications, legal environment, 
and computers. It is this business core that was evaluated using a Primary Trait 
Analysis instrument. 



Literature 

In Assessment Essentials (1999), Palomba and Banta describe Primary Trait 
Analysis (PTA) as one of many assessment techniques that can be useful for classroom 
as well as program assessment. The PTA identifies key factors or traits that are used in 
evaluating an assignment or project, and a standard three- to five-point scoring scale is 
developed for each trait. Each score “is accompanied by an explicit statement that 
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describes performance at that level” (p 164). The higher the score, the more clear, 
complete, and accurate is the student’s performance on that particular trait. Specific 
examples of some of the Primary Traits that emerged through this process and their 
descriptions are provided below. 



Methodology 

The College has a standing assessment committee that meets on a regular basis. 

The committee comprises faculty representatives from each of the four Schools as well 
as the Provost, the Vice-Provost/Director of Institutional Research, a Dean, a 
Department Chair, the Assistant Dean of Student Affairs and the Assistant to the 
President. The College administration and the committee have been strong advocates 
for the assessment process as evidenced by their support for sending committee 
members to national conferences, holding on-campus seminars, and encouraging the 
use of external assessment consultants as appropriate. The SBA has established its own 
assessment committee that works in conjunction with this College-wide committee. 
One-third of the SBA faculty serve on the School’s assessment committee. The SBA 
representative to the College assessment committee serves as an ex-officio member to 
the SBA assessment committee. This committee meets on a regular basis and reports to 
the faculty and administration of the School; all recommendations from the committee 
require approval by the School’s entire faculty and administration. 

The first step in the School’s assessment process was to determine the educational 
outcomes expected by the School of Business Administration. The following outcomes 
were outlined based on institutional mission: 

1 . Students will develop critical thinking, decision making and problem solving skills in 
the application of appropriate business principles and practices. 

2. Students will be proficient in computer applications. 

3. Students will demonstrate verbal and written communication skills. 

4. Students will be aware of the need for developing life long learning skills that will 
prepare them for entry into the business world and/or graduate educational 
opportunities. 

5. Students will meet entry level requirements for employment in business. 

Next, faculty identified the method or methods that would be used to assess these 
outcomes. Based on information that SBA faculty learned during an on-campus 
workshop in Fall 1999, the School selected Primary Trait Analysis as one mechanism 
to measure the desired outcomes. Faculty were very receptive to the PTA process 
outlined by the consultant, and immediate steps were taken to apply this approach. 
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Historically, all business majors are required to take a capstone course, “Administrative 
Policies.” This course offers an opportunity for all students to exhibit the knowledge 
that they have acquired during their matriculation, specifically emphasizing knowledge 
related to the business core. This course provided an appropriate and logical venue in 
which to measure the educational outcomes of SBA majors. 

Data Sources 

The “Administrative Policies” course requires students to complete a 
comprehensive case analysis within a group/team setting. A formal presentation is 
made by the team to other students and faculty from the School. In order to identify the 
traits that would be assessed, all faculty attended the students’ presentations during Fall 
1999. After observing these presentations, individual faculty members developed lists 
of potential primary traits. Early in the Spring Semester, 2000, the SBA assessment 
committee considered this information and compiled a working document of primary 
traits that could be used to assess student outcomes. After careful discussion, the 
committee agreed that the following six primary traits reflect outcomes expected of all 
business majors based on material in the business core: Critical Thinking, Accounting 
and Finance Knowledge, Marketing Knowledge, Use of Visual Aids, Oral Presentation, 
and Written Communication. These six traits were unanimously approved by the 
faculty of the SBA. In addition, they were reviewed and approved by the School’s 
external Advisory Council comprising representatives of local and regional businesses 
who provide feedback to the School. 

Statements were then developed to specify the exact outcomes for each trait that 
would correspond with each of the five levels on the evaluation scale. This represented 
one of the most time-consuming elements of the process since a number of meetings 
were required before the faculty were comfortable that the statements enabled them to 
satisfactorily distinguish various levels of performance. For example, the following 
statement reflects the outcomes for the highest score (5) in Critical Thinking: 

Students exhibited an advanced understanding of Business Principles by 
interpreting information , using appropriate models and techniques (financial 
ratios, strategic management matrices, economic concepts, etc.) and were able to 
logically draw conclusions and make appropriate strategic recommendations . In 
addition, students were able to defend their recommendations . 



Results 

During April 2000, all SBA faculty visited the classes and evaluated the students’ 
team presentations using specific statements such as that shown above. Two to three 
faculty members attended each presentation on a rotating basis. This pilot test provided 
an excellent trial of the traits and statements. Although the results will not be fully 
analyzed for another month, faculty and students have been impacted by the process. 
Clearly, a greater awareness exists of the need for student assessment and the 
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importance of faculty involvement. All faculty who participated learned something as a 
result of the process; they have commented about what they witnessed during the 
presentations and on the level of expectation that the students met. Based on general 
levels of performance in written and oral communication, these traits have already been 
identified as possibilities for improvement. However, no drastic changes will be 
undertaken for several semesters until real trends become clear. 

Overall, there were two major concerns to be addressed in Fall 2000. First, “Did 
the faculty fully understand the process of the primary trait analysis and how to use the 
traits in evaluating student presentations.” In addition, “ Did the students completely 
understand the bases on which they were being evaluated, and were they being 
adequately prepared?” Both the SBA Assessment Committee and the process have 
undergone changes as a result of the initial trial. The two department chairs are now 
responsible for leadership of the Assessment Committee along with a third faculty 
member co-chair. 

The Committee tabulated the results of the primary trait analysis and reached some 
preliminary conclusions. The Committee also surveyed faculty regarding their 
reactions to the process and to suggest possible improvements. The specific questions 
included: 

1. Are we using the relevant primary traits? If not, what would you suggest? 

2. Is the instrument easy to use? If not, what would make it easier for you? 

3. What refinements to these traits would you suggest? 

4. What can be done to help faculty facilitate the process? 

A majority of the faculty responded with valuable suggestions and comments. After 
reviewing their responses, the Committee made minor revisions to the trait scales and 
held a workshop on the course methodology related to the students’ case presentations. 
The workshop clarified the role of faculty in evaluating the presentations and ensured 
that all are fully aware of the evaluation criteria and levels of performance as defined. 

It is critical that faculty focus their evaluation on material that all students should have 
acquired through the business core. 

The course instructor also has revised his course requirements and techniques. For 
example, he has spent additional time explaining the goals of assessment and the 
process to students. They now understand the significance of integrating and relating 
the components of their presentations. Because of the importance placed on visual aids 
and graphics, students are now required to use PowerPoint (learned in the business 
studies core) as the basis of their presentation. Prior to preparing their presentations, all 
students will receive copies of the primary traits and evaluative statements, and they are 
required to maintain a log that lists team meeting dates and activities performed by each 
member of their group. 
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Conclusions and Implications 



This case study suggests that a sound assessment technique can be identified and 
implemented within a short period of time (one semester) given willingness, 
enthusiasm, and commitment by the School committee. Active participation by all 
faculty helped to achieve the buy-in required for this rapid implementation and 
represents a remarkable collaborative effort. In addition, this PTA was initiated using 
existing courses and faculty; no curricular changes were required, and no additional 
funds were necessary. The PTA promises to provide valuable information that will 
enable the School to improve its programs and enhance the overall performance of 
business majors. It is hoped that, through its early success, this process will become a 
model for other departments and schools at the College. 





WEST LIBERTY STATE COLLEGE 
SCHOOL OF BUSINESS ADMINISTRATION 
ASSESSMENT 2000 



EDUCATIONAL OUTCOMES 

1 . Students will develop critical thinking, decision making and problem 
solving skills in the application of appropriate business principles and 
practices. 

2. Students will be proficient in computer applications. 

3. Students will demonstrate verbal and written communication skills. 

4. Students will meet entry level requirements for employment in business.. 

PRIMARY TRAITS 
Critical Thinking 

Accounting and Finance 

Marketing 

Visual Aids 

Oral Presentation Skills 
Written Communication Skills 
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WORKSHEET - PRIMARY TRAIT SCALES 



CLASS: MGT 498 - ADMINISTRATIVE POLICIES 
GROUP PRESENTATION/COMPANY NAME: 



EVALUATOR: 



PRIMARY TRAITS 

Rating Scale 
5 4 3 2 1 

A. Critical Thinking _____ 

B. Accounting/Finance _____ 

C. Marketing _____ 

D. Visual Aids _____ 

E. Oral Presentation _ _ _ _ 



F. Written Communication 



SCHOOL OF BUSINESS ADMINISTRATION 
ASSESSMENT 
CRITICAL THINKING 



5. Students exhibited an advanced understanding of Business principles by 
interpreting information , using appropriate models and techniques (Financial 
Ratios, Strategic Management Matrices, Economic Concepts, etc.) and were 
able to logically draw conclusions and make appropriate Strategic 
recommendations . In addition, students were able to defend their 
recommendations . 

4. Students exhibited an advanced^ understanding of Business principles by 
interpreting information, using appropriate models and techniques (Financial 
Ratios, Strategic Management Matrices, Economic Concepts, etc.) and were 
able to logically draw conclusions and ntake appropriate Strategic 
reconunerulatipns. Students were unable to defend their recommendations. 

3. Students exhibited an understanding of Business principles by interpreting 
infonnatipn, using appropriate models and techniques (Financial Ratios, 
Strategic Management Matrices, Economic Concepts, etc.). Students were able 
to draw conclusions (not necessarily logical and make Strategic 
recommendations. Students were unable to defend their recommendations. 

2. Students exhibited some understanding of Business principles but failed to 
properly interpret information or apply business models or techniques (Financial 
Ratios, Strategic Management Matrices, Economic Concepts, etc.). Students 
failed to draw conclusions^ op ntake Strategic recommendations. 

1. Students exhibited no understanding of Business principles. Students did not 
interpret^ infonnatipn or apply business models and technigues (Financial Ratios, 
Strategic Management Matrices, Economic Concepts, etc.) Students failed to 
draw conclusions or make Strategic recommendations. 
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SCHOOL OF BUSINESS ADMINISTRATION 
ASSESSMENT 

ACCOUNTING AND FINANCE 



5. Students exhibited an advanced understanding of Accounting and Financial 
concepts by ayylyins and interpreting appropriate techniques, models and data 
(Income Statements, Balance Sheets, Financial Ratios, etc.) and were able to 
logically draw conclusions and make appropriate Financial recommendations. In 
addition, students were able to defend their recommendations. 

4. Students exhibited an advanced understanding of Accounting and Financial 
concepts by ayylyins, and interpreting apgrogriate techniques, models and data 
(Income Statements, Balance Sheets, Financial Ratios, etc.), and were able to 
logically draw conclusions and make ayyroyriate Financial recommendations. 
Students were unable to defend their recommendation^ 

3. Students exhibited an understanding of Accounting and Financial concepts by 
ayylyine and interpreting ayyroyriate techniques, models and data (Income 
Statements, Balance Sheets, Financial Ratios, etc.). Students were able to draw 
conclusions (not necessarily logical ) and make Financial recommendations. 
Students were unable to defend their recommendation s. 

2. Students exhibited some understanding of Accounting and Financial concepts 
but failed to yroyerly interpret the ayyroyriate techniques, models and data 
(Income Statements, Balance Sheets, Financial Ratios, etc.). Students failed to 
draw conclusions or make Financial recommendations. 

1. Students exhibited no understanding of Accounting and Financial concepts. 
Students were unable to ayyly or interpret the ayyroyriate technique s. models 
and data (Income Statements, Balance Sheets, Financial Ratios, etc.). Students 
failed to draw conclusions or make Financial recommendations. 
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SCHOOL OF BUSINESS ADMINISTRATION 
ASSESSMENT 
MARKETING 



5. Students exhibited an advanced understanding of the principles of Marketing 
by interpreting information about target market selection and the development 
of product, distribution, price, and promotion and were able to logically draw 
conclusions and make appropriate Marketing recommendations. In addition, 
students were able to defend their recpniniendations. 

4. Students exhibited an advanced understanding of the principles of Marketing 
by interpreting infonnation about target market selection and the development 
of product, distribution, price, and promotion and were able to logically draw 
conclusions^ and niake appropriate Marketing recommendations. Students were 
unable to defend their reconunendations . 

3. Students exhibited an understanding of the principles of Marketing by 

interpreting infonnation about target market selection and the development of 
product, distribution, price, and promotion. Students were able to draw 
conclusions (npl necessarily logical ) and niake Marketing recommendations. 
Students were unable to defend their recommendations. 

2. Students exhibited some understanding of principles of Marketing but failed to 
properly interpret, information about target market selection and the 
development of product, distribution, price, and promotion. Students failed to 
draw conclusion s or niake Marketing r ecpnunendatipns . 

1. Students exhibited no understanding of the principles of Marketing. Students 
did nop interpret, infonnation about target market selection and the 
development of product, distribution, price, and promotion. Students failed to 
draw conclusions or niake Marketing recommendations. 
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SCHOOL OF BUSINESS ADMINISTRATION 
ASSESSMENT 
VISUAL AIDS 



5. Students exhibited an advanced, knowledge of the principles of Visual 

Presentations Techniques and utilized up-to-date computer, generated, visuals. 
The visual aids enhanced the viewer’s understanding of the material being 
presented. The presentation was well rehearsed; the visuals were an integral 
part of the presentation. 

4. Students exhibited an advanced knowledge of the principles of Visual 
Presentation Techniques but did not utilize up-to-date computer generated 
visuals. The visual aids enhanced the viewer’s understanding of the material 
being presented. The presentation was well rehearsed; the visuals were an 
integral part of the presentation. 

3. Students exhibited some knowledge of the principles of Visual Presentation 
Techniques and utilised up-to-date computer. generated, visuals. The visual aids 
did little to enhance the viewer’s understanding of the material being 
presented. The presentation was not well rehearsed. 

2. Students exhibited limited knowledge of the principles of Visual Presentation 
Techniques and did not, utilize up-to-date computer generated, visuals. The visual 
aids did little to enhance the viewer’s understanding of the material being 
presented. The presentation was not well rehearsed. 

1. Students exhibited no knowledge, of the principles of Visual Presentation 

Techniques. The visual aids did nothing to enhance the viewer’s understanding 
of the material being presented. 
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SCHOOL OF BUSINESS ADMINISTRATION 
ASSESSMENT 

ORAL PRESENTATION SKILLS 



5. Students exhibited an advanced understanding Oral Presentation Skills by 
utilizing appropriate delivery methods such as speaking from notes, using simple 
language, providing frequent summaries of key points, using appropriate voice 
quality, maintaining effective audience eye contact, providing a strong and 
effective opening and closing, and effectively handling the auestion-and-answer 

sess ion. 

4. Students exhibited an advanced understanding of Oral Presentation Skills by 
utilizing appropriate delivery, methods such as speaking from notes, using simple 
language, providing frequent summaries of key points, using appropriate voice 
quality, maintaining effective audience eye contact, and providing a strong and 
effective opening and closing. Students were unable to effectively handle the 
question = and : answer session. 

3. Students exhibited an understanding of Oral Presentation Skills by utilizing 
appropriate delivery methods such as speaking from notes, using simple 
language, providing frequent summaries of key points, using appropriate voice 
quality, maintaining effective audience eye contact, but did not provide a strong 
and elective opening and closing. Students were also unable to effectively 
handle the auestion-and-answer session. 

2. Students exhibited some understanding of Oral Presentation Skills but poorly 
utilized appropriate delivery methods such as using simple language, frequent 
summaries of key points, and appropriate voice quality. Students failed to 
maintain effective audience eye contort and did not provide a strong and effective 
opening and closing Students were also unable to effectively handle the 
questurn = and answer session. 

1. Students exhibited no understanding of Oral Presentation Skills. Students 
Jaded to utilize appropriate delivery methods such as using simple language, 
frequent summaries of key points, and appropriate voice quality. Students also 
failed to maintain effective audience eye contact and did not provide a strong and 
effective^ opening and closing. Students were also unable to effectively handle the 
guestwn-and-answer session. 
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SCHOOL OF BUSINESS ADMINISTRATION 
ASSESSMENT 

WRITTEN COMMUNICATION SKILLS 



5. Students exhibited an advanced understanding of Written Communications Skills by 
utilizing appropriate writing techniques for reports such as sequencing information in a 
logical order, with a clearly defined purpose, an appropriate introduction which 
explains what and why, a body that explains how, where or how much, and developing 
conclusions that support the body of the report. Students utilized appropriate report 
format that is easy to read, including appropriate graphics and headings which lead 
the reader through the information in a consistent manner. Students also utilized an 
appropriate tone t convincing and precise language , and simple sentences utilizing 
correct spelling and grammar . 

4. Students exhibited an advanced understanding of Written Communications Skills by 
utilizing appropriate writing techniques for reports such as sequencing information in a 
logical order, with a clearly defined purpose, an appropriate introduction which 
explains what and why, a body that explains how, where or how much, and developing 
conclusions that support the body of the report. Students utilized appropriate report 
format that is easy to read, including appropriate graphics and headings which lead 
the reader through the information in a consistent manner. Students also utilized an 
appropriate tone . convincing and precise language . and simple sentences utilizing 
correct spelling and grammar . Students were unable to effectively present the data in a 
consistent manner . 

3. Students exhibited an understanding of Written Communications Skills by utilizing 
appropriate writing techniques for reports such as sequencing information in a logical 
order, with a clearly defined purpose, an appropriate introduction which explains 
what and why, a body that explains how, where or how much, and developing 
conclusions that support the body of the report. Students utilized appropriate report 
format that is easy to read, including appropriate graphics and headings which lead 
the reader through information in a consistent manner. Students also utilized an 
appropriate tone , convincing and precise language . and simple sentences utilizing 
correct spelling and grammar . Students were unable to effectively present the data in a 
consistent manner. 

2. Students exhibited some understanding of Written Communications Skills but failed to 
utilize appropriate writing techniques for reports . Students failed to use an appropriate 
report format that is easy to read, and did not utilize headings which would lead the 
reader through the information in a consistent manner. 

1. Students exhibited no understanding of Written Communications Skills. Students did 
not utilize appropriate writing techniques for reports. Students failed to utilize 
appropriate tone % convincing and precise language . and simple sentences utilizing 
correct spelling and grammar. 

10/23/00 
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Introduction 

This study of remedial English course at West Chester University was undertaken at 
the request of the Developmental Education Task Force, which Dr. Skerl chaired and 
which had representatives from the English Department, the Mathematics Department, 
and developmental education support services. One of the charges was for the Task 
Force to review the structure and effectiveness of remedial English and Mathematics 
courses and to propose to the Provost alternative structures if warranted by the review. 

West Chester University’s policy indicates that, “Placement in the appropriate 
composition course is determined by the score on the SAT and/or by performance on a 
placement test administered by the Department of English.” (p. 33, West Chester 
Undergraduate Catalog, 1999-2000). SAT Verbal (SAT-V) scores and an optional 
placement writing challenge exam are used to determine whether students must first be 
placed in a zero-level remedial composition course before being permitted to enroll in 
100-level English courses, which are the college-level required courses. The cutoff score 
for remedial English placement was 450 SAT-V before the recentering, and 500 after it. 
Students must earn a grade of C- or better in order to pass the zero-level remedial courses 
before they are permitted to enroll in the 100-level courses. West Chester University 
requires all students to take two college-level composition courses as part of their general 
education requirements. 

Although a very large percentage of entering freshmen at WCU are placed in these 
courses (about one-third in English and fourteen percent in Mathematics remedial 
programs,) there had been no comprehensive evaluation of the effectiveness of these 
courses since their inception over 20 years ago. Therefore, the Task Force asked Dr. Zhai 
from Office of Planning & Analysis to study the impact of remedial programs. Results 
and analyses about remedial Mathematics were presented at the 26th NEAIR conference. 
Since then we updated our initial studies of remedial English course. Results and 
analyses are presented here. 





As pointed out by Weissman, Bulakowski and Jumisko (1997): “The purpose of 
remedial courses is to enable students to gain the skills necessary to complete college- 
level courses and academic programs successfully.” Based on these guidelines, this study 
tried to examine the following issues: (1) To what extent are the remedial English courses 
effective in preparing students for their college-level required English courses? (2) To 
what extent do the remedial English courses contribute to students’ academic success as 
shown by their retention and graduation rates? 

Methodology 

Data 

Student course grades for the remedial (ENG 020) and two other required college- 
level English courses (ENG 120 & ENG 121), their SAT-V scores, admission type, 
enrollment status and graduation records were used in this study. Data were taken from 
the University’s historical snapshots and the Student Flow Models maintained by the 
Office of Planning & Analysis. This study covers the period from Fall 1992 to Spring 
2000. 

Selection of the Comparison Group (Control Group ) 

One of the major challenges facing the evaluation of remedial course impact in this 
four-year public institution is the lack of student comparison groups due to the remedial 
course placement policy adopted by the university. For this study it is assumed that, in 
order for a remedial program to be judged effective, it ought to help some students 
succeed who otherwise would most likely fail their college-level coursework. It was also 
assumed that, if the English remedial program can help some under-prepared students to 
succeed, it would fulfill its function. 

In order to ensure reasonably informative comparisons, the control or comparison 
group used for this study were those students who scored no more than 50 points higher 
than the SAT-V cutoff score for placement into the remedial program. The cutoff score 
for remedial English was 450 before the recentering of SAT in fall 1996 and 500 after the 
recentering. As a result, the placement score for the Control Group was SAT-V above 
the cutoff score, but below or equal to 500 (550 after the recentering). 

Due to WCU’s policy, an entering student with SAT-V below 450/500 may be placed 
out of remedial program if that student takes the English placement test and successfully 
passes it. A student may also be placed out of ENG 020 if that student has Advanced 
Placement credit or transferred credits from comparable English composition courses. In 
the forthcoming analysis, this group of students will be separated from the remedial and 
the comparison group. 
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Definition of Terms 



Student Groups: 

• remedial group - students who took at least one remedial English course during their 
matriculation in the University 

• placed-out group - students with SAT-V below the cut-off score (450/500) who were 
placed out of the remedial program by taking a placement test given by the English 
Department 

• Control Group - students whose SAT-V were high enough to place them out of the 
remedial program but lower than 500 (550 after the recentering) 

• college-ready - students whose SAT-V scores were higher than 500/550 

• no-SAT-V - students with no SAT-V (transfer and non -traditional students) 

Admission Status: 



West Chester University admits students in four categories: regular admission and 
three categories of special admissions for those students who do not meet the criteria for 
regular admission: Academic Development Program Act 101, Academic Development 
Program non-Act 101, and Special Admit Motivational. The minimum qualifications for 
each category are as follows: 

• Regular Admit - Academic program continued into senior year; combined SAT of 
1000; High School Rank 50%; and Honors or AP classes a plus 

• Academic Development Program Act 101 (ADP Act 101 ) - Verbal SAT 380; Math 
SAT 340; High School Rank 40%; and GPA 2.0 

• Academic Development Program Non-Act 101 (ADP Non-Act 101) - Similar as 
ADP Act- 101, but without special financial assistance 

• Special Admit Motivational (Special Admit) - Verbal SAT 480; Math SAT 450; 
High School Rank 60%; and GPA 2.7 

Outcome Measures 

Three major outcome measures were employed to assess the impact of the remedial 
program. They are: (1) student performance in college-level English composition 
courses; (2) second-year retention rates and (3) six-year graduation and retention rates. 
Outcome measures were collected and compared between remedial students and students 
in the Control Group. 

It is NOT the intention of this study to compare developmental students with other 
college-ready students. Information concerning other students was included in this study 
for reference only. 
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Statistics 



Chi-square statistics were used to compare student course passing rates between 
remedial students and the Control Group. A grade of C- or better was considered a 
passing grade. One-way ANOVA was used to detect course performance differences on 
college-level English work. Due to the large sample size (4,388 records for ENG 020, 

1 1,247 for ENG 120 and 14,305 for ENG 121), all statistical analyses yielded significant 
statistical results even when the magnitude of the difference was of little practical 
concern (for example a GPA of 2.69 vs. 2.83). As a result, statistical results were not 
reported. Instead, emphases were placed on the practical application of the findings 
when pertinent. Detailed statistical results are available upon request. 

Results and Analyses 



Course-Takers 

From fall 1992 to spring 2000, there were 4,060 students who took ENG 020 and 328 
of them had to repeat the course at least once. The majority of ENG 020 course takers 
were first-time, full-time degree-seeking students. Taking the Fall 1999’s ENG 020 class 
for example: There were 564 students enrolled in the course. About 98% of them were 
first-time, degree seeking students. Of the 564 students, 59% were Regular Admits, 5%, 
ADP-Act 101, 7%, ADP-Non Act, and 27%, Special Admits. Table 1 tabulates the class 
profile for Fall 1999. 



Table 1. Summaries of ENG 020 Class Profile for Fall 1999 



ENG 020 Fall 1999 Class 


995 Freshman Cohort 






% within the 




# by Adm 




Admission Status 


N 


Class 


SAT-V 


Type 


% taking 020 


ADP 


29 


5.14 


408 


48 


60.42 


ADP-Non Act 101 


43 


7.62 


417 


77 


55.84 


Regular Admit 


334 


59.22 


469 


1,374 


24.31 


Special Admission 


150 


26.60 


448 


201 


74.63 


Regular Admit (Transfer) 


3 


0.53 








Admission Info Missing 


5 


0.89 










564 






1,700 





Remedial Student Course-Taking Patterns 

Student course-taking pattern tracking showed that the majority of the students took 
ENG 020 in fall. If he/she passed the course by earning a grade of C- or better, he/she 
would proceed to take ENG 120 in spring and ENG 121 the following fall. If a student 
failed to pass ENG 020, he/she would usually repeat it in spring and then moved on to 
take ENG 120 the following fall, if he/she passed ENG 020. Table 2 provides a brief 
summary of the course passing status in the past 8 years. The total number in Table 2 is 
not unduplicated headcount. If a student took ENG 020 twice, once with a grade below 
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C- and once with a grade C- or better, that student will be counted once in the Pass and 
once in the Fail to Pass. As shown in Table 2, the success rate for ENG 020 was about 
87%. 



Table 2. Summaries of ENG 020 Student Course Grade Distribution 







Frequency 


Percent 


Valid 

Percent 


Cumulative Percent 


Valid 


Pass 


3818 


87.0 


87.0 


87.0 




Fail to Pass 


450 


10.3 


10.3 


97.3 




Withdraw 


120 


2.7 


2.7 


100.0 




Total 


4388 


100.0 


100.0 




Total 




4388 


100.0 







After passing ENG 020, about 80% (3235/4060) of the remediated students proceed to 
take ENG 120. 

Remediated Student Course Performance in ENG 120 

In order to see how remedial English helped preparing the students for their college- 
level course work, we first took a look at student course performance in ENG 120. Table 
3 presents student course completion rates by the five student groups. 



Table 3. Summaries of ENG 120 Student Grade Distribution by Student Comparison Groups 









ENG 120 Course Passing Grade 










Pass (C- or 


Fail to Pass 












Better) 


(Below C-) 


Withdraw 


Total 


Student 


Remediated Students 


Count 


3235 


204 


128 


3567 


Comparison 

Groups 




% within Student 
Comparison Groups 


90.7% 


5.7% 


3.6% 


100.0% 




PlacedOut - No Remedial, SATV 


Count 


1271 


92 


37 


1400 




below 450/500 (Placed-out) 


% within Student 
Comparison Groups 


90.8% 


6.6% 


2.6% 


100.0% 




Control - No Remedial, SATV 


Count 


2756 


176 


94 


3026 




>=450/500 and <500/550 


% within Student 
Comparison Groups 


91.1% 


5.8% 


3.1% 


100.0% 




College-Ready, SATV >500/550 


Count 


1743 


124 


65 


1932 






% within Student 
Comparison Groups 


90.2% 


6.4% 


3.4% 


100.0% 




No SATV 


Count 


1028 


144 


150 


1322 






% within Student 
Comparison Groups 


77.8% 


10.9% 


1 1 .3% 


100.0% 


Total 




Count 


10033 


740 


474 


11247 






% within Student 
Comparison Groups 


89.2% 


6.6% 


4.2% 


100.0% 
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According to Table 3, 90.7% of remediated students who took ENG 120 successfully 
passed the course, compared with 90.8% in the placed-out group, 91.1% in the Control 
Group, 90.2% in the college-ready group, and 77.8% in the no-SAT-V group. A study 
of the means of student course grades for the various groups in Table 4 shows that not 
only the passing rates between the remediated and the control groups were very 
comparable, the means were also very close. The mean course grade was 2.70 for the 
remediated students, 2.64 for the placed-out group, 2.74 for the Control Group, 2.863 for 
the college-ready group, and 2.73 for the no-SAT-V group. 



Table 4. Comparisons of Student Course Performance in Eng 120 

ENG 120 Course Grade 



Student Comparison Groups 


Mean 


N 


Std. Deviation 


Remediated Students 


2.6969 


3439 


.7968 


PlacedOut - No Remedial, SATV 
below 450/500 (Placed-out) 


2.6390 


1363 


.8390 


Control - No Remedial, SATV 
>=450/500 and <500/550 


2.7474 


2932 


.8427 


College-Ready, SATV >500/550 


2.8647 


1867 


.8945 


No SATV 


2.7311 


1172 


1.1466 


Total 


2.7361 


10773 


.8780 



Remediated Student Course Performance in ENG 121 

An examination of remediated students’ performance in ENG 121 revealed similar 
results as found in ENG 120. Tables 5 & 6 exhibits how remediated students performed 
in ENG 121 compared with the Control Group. 

According to Table 5, the passing rates for the remediated and the Control Group 
were very close: 82.3% for the former and 84.2 for the latter. In general about 81.3% of 
students who took ENG 121 pass the course. Results in Table 6 reveal that even though 
remediated students tend to have a similar passing rate as their non-remediated peers, 
their individual grades tend to be lower than those earned by their peers. For example, 
the mean grade for the remediated group was 2.54, as shown in Table 6, while the mean 
grades for the control and college-ready groups were 2.66 and 2.81 respectively. 



244 

O 

ERLC 



238 



Table 5. Summaries of ENG 121 Student Course Grade Distribution by Student Comparison Groups 









ENG 121 Course Passing Grade 










Pass (C- or 


Fail to Pass 












Better) 


(Below C-) 


Withdraw 


Total 


Student 


Remediated Students 


Count 


2405 


324 


195 


2924 


Comparison 

Groups 




% within Student 
Comparison Groups 


82.3% 


11.1% 


6.7% 


100.0% 




PlacedOut - No Remedial, SATV 


Count 


1406 


238 


92 


1736 




below 450/500 (Placed-out) 


% within Student 
Comparison Groups 


81.0% 


13.7% 


5.3% 


100.0% 




Control - No Remedial, SATV 


Count 


1846 


228 


118 


2192 




>=450/500 and <500/550 


% within Student 
Comparison Groups 


84.2% 


10.4% 


5.4% 


100.0% 




College-Ready, SATV >500/550 


Count 


1751 


222 


175 


2148 






% within Student 
Comparison Groups 


81.5% 


10.3% 


8.1% 


100.0% 




No SATV 


Count 


4220 


543 


542 


5305 






% within Student 
Comparison Groups 


79.5% 


10.2% 


10.2% 


100.0% 


Total 




Count 


11628 


1555 


1122 


14305 






% within Student 
Comparison Groups 


81.3% 


10.9% 


7.8% 


100.0% 



Table 6. Comparisons of Student Course Performance in Eng 121 

ENG 121 Course Grade 



Student Comparison Groups 


Mean 


N 


Std. Deviation 


Remediated Students 


2.5418 


2729 


.9945 


PlacedOut - No Remedial, SATV 
below 450/500 (Placed-out) 


2.4322 


1644 


1.0181 


Control - No Remedial, SATV 
>=450/500 and <500/550 


2.6558 


2074 


1.0255 


College-Ready, SATV >500/550 


2.8083 


1973 


1.1010 


No SATV 


2.7450 


4763 


1.0911 


Total 


2.6593 


13183 


1.0611 



Results from this analysis confirm the findings by Weissman, Silk and Bulakowski 
(1997), who found that although the average GPA for the remediated students was not as 
high as that of college-ready students, remediated students performed at above a C 
average in their college-level courses. For our study, we found that our remediated 
students averaged a B- in ENG 1 20, just as the rest of their peers. Remediated students 
tend to earn C+ in ENG 121 compared with an average of B- for the control and the 
college-ready groups. Since the University allows students with high SAT-V to skip 
ENG 120 by taking ENG 121 directly, we saw more college-ready students in the 
analysis of ENG 121 than in ENG 120. 
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Remediated Student Second-Year Retention Rates 



The second measure used to assess the impact of remedial English course was student 
second-year retention rates. In order to get more accurate assessment of the impact that 
the remedial English program had on student persistence and graduation rates, only first- 
time, full-time degree-seeking remedial student retention and graduation rates were used. 
As a result, the following comparisons and analyses will be based on cohort data, instead 
of student course class. 



Table 7 summarizes the percentage of students taking remedial English. Table 8 
presents the second-year retention rates when the same cohort were regrouped according 
to if they had taken remedial English or not. 



Table 7. Summaries of First-time, Full-time, Degree-seeking Students Taking 
Remedial English Fall 1992 - 1999 



1st Fall Enrolled 


Cohort 


Taking Remedial ENG 


NonRemedial 


Year 


N 


% 


N 


% 


Total Cohort 


1992 


388 


28.53 


972 


71.47 


1,360 


1993 


422 


30.89 


944 


69.11 


1,366 


1994 


503 


37.09 


853 


62.91 


1,356 


1995 


448 


32.58 


927 


67.42 


1,375 


1996 


507 


35.04 


940 


64.96 


1,447 


1997 


536 


34.12 


1,035 


65.88 


1,571 


1998 


507 


31.30 


1,113 


68.70 


1,620 


1999 


544 


32.00 


1,156 


68.00 


1,700 


Multi-year Average 


32.68 




67.32 





According to Table 7, in fall 1992, there were 1,360 students enrolled as first-time, 
full-time, degree-seeking students. Of them, 388 (28.53%) took ENG 020 that fall. 
Table 8 revealed that the second-year retention rates for the 1992 cohort were: 82% for 
non-remedial students and 89.89% for the remediated students. For the 1993 cohort, the 
rates were 80% for non-remedial course takers and 91% for remediated students. 
Generally speaking, remediated students seem to have higher second-year retention rate 
than the rest. One factor we will need to consider is that in WCU, ADP students are 
committed to enroll for two years. 
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Table 8. Comparisons of Second-Year Retention Rates Between Remediated and 
Non-Remediated First-time, Full-time, Degree-seeking Students 

2nd Fall Retention Rate 



Remedial NonRemedial Total 



Cohort 


N 


% Retained 


N 


% Retained 


N 


% Retained 


1992 


360 


92.78 


770 


79.22 


1130 


83.09 


1993 


388 


91.94 


748 


79.24 


1136 


83.16 


1994 


411 


81.71 


671 


78.66 


1082 


79.79 


1995 


364 


81.25 


763 


82.31 


1127 


81.96 


1996 


400 


78.90 


770 


81.91 


1170 


80.86 


1997 


446 


83.21 


853 


82.42 


1299 


82.69 


1998 


417 


82.25 


935 


84.01 


1352 


83.46 


1999 


468 


86.03 


948 


82.01 


1416 


83.29 


Multi-year Average 


84.76 




81.22 







Table 9 gives the second-year student retention rates by the University’s admission 
types. According to Table 9, both ADP and Special Admit students have comparable 
second-year retention rates as the Regular Admit. As a result, the higher second-year 
retention rate for the remediated students as shown in Table 8 might be due to those 
students’ enrollment commitment as well. More evidence is needed to assess remedial 
English program’s impact on the retention issue. 

Table 9. Second- Year Retention Rates For First-time, Full-time, Degree-seeking Student 



Cohorts 



Fall Cohort 


Regular Admit 


ADP-ACT 101 


ADP-Non ACT 101 


Special Admit 


1992 


83.5% 


72.2% 


82.9% 


85.3% 


1993 


82.9% 


76.8% 


90.9% 


85.6% 


1994 


78.9% 


83.6% 


87.9% 


82.4% 


1995 


82.4% 


77.2% 


80.6% 


80.7% 


1996 


80.5% 


81.8% 


89.1% 


80.1% 


1997 


82.5% 


90.7% 


83.1% 


81.7% 


1998 


82.9% 


89.7% 


89.5% 


83.7% 


1999 


82.4% 


83.3% 


94.8% 


85.1% 


Multi-Year Average 


82.0% 


81.9% 


87.4% 


83.1% 



Remediated Student Six-Year Graduation and Retention Rates 

The third measure used to assess the remedial English program’s impact is the six- 
year retention and graduation rate. Table 10 present comparisons of the six-year retention 
rates for students with or without taking remedial coursework. The six-year retention and 
graduation rates were based on three cohorts from 1992 to 1994. 
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Table 10. Six-Year Graduation and Retention Rates for Fall 1992 - 94 First-Time, FuII- 
__ == __Tmiej n De£ree-Seeldn£^tudentCohorteasofFall2000^^^^^^^^^^^ 



6-year Retention & Graduation Rates 



Remediated Students 


No ENG 020 


Cohort 

Year 


Graduated 


Enrl 

(7thFall) 


Grad+Enrl 


Graduated 


Enrl 

(7thFall) 


Grad+Enrl 


1992 


245 


16 


261 


495 


14 


509 


1993 


230 


16 


246 


488 


26 


514 


1994 


220 


8 


228 


398 


22 


420 








Retention 






Retention 




% 


% 


Rate 


% 


% 


Rate 


1992 


63.1 


4.1 


67.3 


50.9 


1.4 


52.4 


1993 


54.5 


3.8 


58.3 


51.7 


2.8 


54.4 


1994 


43.7 


1.6 


45.3 


46.7 


2.6 


49.2 


Average 


53.8 


3.2 


57.0 


49.8 


2.3 


52.0 



Table 10 shows that the six-year retention and graduation rate for remediated students 
was 57%, about 5% higher than those who didn't take ENG 020. 



Six-year retention rates of remediated students were also compared with those of 
other student groups and the results were tabulated in Table 1 1. Those rates were also 
based on the averages of Fall 1992 - 94 cohorts. For example, the six-year retention and 
graduation rate was 56.7% for the Regular Admit, 55.7% for the Special Admit, 29.8% 
for ADP Act 101, and 48.3% for ADP Non-Act 101. The general six-year retention rate 
was 54.9% for the University. 



Table 11. Comparisons of Six-Year Graduation and Retention Rates Between Remediated 



Students and Other Student Groups 



Admission Type 


Retention 


Graduation 


Total Ret. & Grad. 


Regular Admit 


2.4 


54.3 


56.7 


ADP-Act 101 


5.2 


24.6 


29.8 


ADP-Non Act 101 


1.8 


46.5 


48.3 


Special Admit 


2.1 


53.6 


55.7 


Remediated 


3.2 


53.8 


57.0 


Non-Remedial 


2.3 


49.8 


52.0 


University Total 


2.5 


52.4 


54.9 


National Average 25 (CSRDE, 2000) 
Moderately selective 26 
Selective 


44.2% 

53.6% 





25 Institution size 5,000 - 17,900 

26 Moderately Selective SATs 900 - 1044; Selective SATs 1045 - 1 100 
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Table 1 1 also gives a national average student retention rate as reported by the 
Consortium for Student Retention Data Exchange (CSRDE), in May 2000. CSRDE 
reported that the national averages for six-year retention and graduation rates were 53.6% 
for selective institutions and 44.2% for moderately selective institutions. WCU is one of 
the “Moderately Selective Institutions” based on CSRDE’ s criteria. According to 
CSRDE, not only West Chester University’s general six-year retention and graduation 
rates were above the national norm (54.9% vs. 44.2%), its remediated first-time degree- 
seeking students’ six-year retention rate was even higher than the University’s average 
(57% vs. 54.9%). The retention rate index for Selective Institutions was 53.6%, 
according to CSRDE. West Chester University’s 6-year student retention and graduation 
rates was 54.9%, slightly higher than 53.6%. 

Conclusions & Recommendations 

Based on the findings from this study, we concluded: 

1. ENG 020 prepares students effectively for ENG 120 and 121. 

2. ENG 020 supports students’ overall academic success, as measured by retention 
and graduation. 

3. The academic success of ENG 020 students, and their strong showing in 
subsequent writing courses, suggests that the placement procedure is appropriate. 

Our findings and conclusions led to the following recommendations pertaining to the 
English remedial course: 

1 . Given the success of English 020, a major overhaul is not necessary; however. 
Task Force members believe that the program can be improved and updated 
according to current best practices. The Task Force recommends that the English 
Department consider the following alternative structures for its developmental 
composition program: smaller classes; two-semester courses; an expanded 
Writing Center which works more closely with instructors and students in ENG 
020; studio courses; more frequent class meetings. 

2. A special information meeting should be scheduled as part of summer Orientation 
for students placed in zero-level courses and their parents. English Department 
representatives would have the opportunity to explain to students and their 
parents: the WCU English placement policy and procedure, the educational 
rationale, and — most important — the benefits of placement in ENG 020. 

3. Communication with our feeder high schools about our academic standards and 
placement criteria for English should be improved via information on the 
University website, distribution of a brochure/information sheet to teachers and 
school officials, and meetings between Admissions staff and school officials. 
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NEAIR 27 th Annual Meeting 
Saturday, November 4 th , 2000 



Conference Program 



1:00 - 5:00 pm 

Duquesne Room - Lower Lobby 


Conference Registration 


2:00 - 5:00 pm 
Forbes - Lower Level 
Karen Bauer 

Assistant Director of Institutional 
Research and Planning 
University of Delaware, 

NEAIR Past-President 


Newcomers to Institutional Research, Part 1 

This workshop is designed for new practitioners who 
engage in IR activities. This workshop addresses key 
components of IR including defining critical issues for 
institutional research, identifying sources of data, 
developing fact books and other reports, and conducting 
effective survey research for assessment and evaluation. 
The main focus is a presentation of general concepts and 
practical strategies for the implementation of continued 
development of effective IR at many schools, regardless of 
size or type. 

Pre-Conference Workshop 


2:00 - 5:00 pm 
Stanwix - Lower Level 
Mary Ann Coughlin 
Professor of Research & Statistics 
Springfield College, 

NEAIR Treasurer 


Statistics for Institutional Research 

Basic ideas in statistics will be covered in a way that is 
useful as an introduction or as a refresher to statistics. 
Descriptive statistics, sampling and probability theory as 
well as the inferential methods of chi-square, t-test and 
Pearson’s r will be covered. May be taken with or without 
the follow-up advanced workshop. 

Pre-Conference Workshop 


2:00 - 5:00 pm 
Heinz - Lower Level 
William E. Knight 

Director of Planning and Institutional 
Research 

Bowling Green State University 


Path Analysis for Beginners 

This workshop will introduce path analysis in a hands-on 
and straightforward manner, targeting the areas of 
assessment and enrollment management research. Data 
from the presenters’ institutions will be utilized and detailed 
handouts provided. Attendees with laptops and copies of 
SPSS AMOS 4.0 are encouraged to bring them, but not 


Corby A. Coperthwaite 

Director of Planning, Research and 

Assessment 

Manchester Community College 


required. 

Pre-Conference Workshop 


2:00 - 5:00 pm 

Board Room - Lower Level 
Anne Marie Delaney 
Director of Institutional Research 
Babs on College, 

NEAIR President-Elect 


Research Design Ideas for Institutional Researchers 

The primary goal of this workshop is to enhance 
institutional researchers’ capacity to produce policy relevant 
studies for planning and decision-making. Specific 
objectives include enabling participants to translate data 
into information; to transform reporting into research; and 
to prepare methodologically sound, practically useful 
research reports for their institutions. The workshop will 
demonstrate how the institutional researcher can use 
principles of research design and selected research 
techniques to transform data collection activities into 
decision-oriented research projects. 

Pre-Conference Workshop 


6:00 - 7:00 pm 

King's Garden North and South - 
Mezzanine Level 


Earlv Bird Reception sponsored bv SPSS 
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NEAIR 27 th Annual Meeting 
Sunday, November 5 th , 2000 



Conference Program 



8:00 - 4:30 pm 

Ballroom 4 - Mezzanine Level 


Conference Registration 


9:00 - noon 

Rivers - Mezzanine Level 

J. Fredericks Volkwein 

The Pennsylvania State University, 

NEAIR President 


The Three Stages of Enrollment Management 

Enrollment management is a component of institutional 
effectiveness and quality control. At the first stage, 
enrollment management includes attracting, admitting, 
and enrolling students. This is the set of admissions 
activities that campus managers traditionally think of as 
constituting the core of Enrollment Management. At the 
second stage lies activities that surround the new student 
experience — activities that ensure the student's 
successful introduction and integration into the 
institution. At the third stage, enrollment management 
focuses upon the quality and totality of the student 
experience — experiences and factors producing high 
academic performance, student persistence to degree 
completion, and success in the world beyond the campus. 
Pre-Conference Workshop 


9:00 - noon 

Brigade - Mezzanine Level 
Karen Bauer 

Assistant Director of Institutional 
Research and Planning, University of 
Delaware, NEAIR Past-President 


Newcomers to Institutional Research* Part 2 
Continuation; Part 1 is a pre-requisite. 
Pre-Conference Workshop 


9:00 - noon 

Traders - Mezzanine Level 
Mary Ann Coughlin 
Professor of Research & Statistics 
Springfield College, 

NEAIR Treasurer 


Advanced Statistics for Institutional Research 

This workshop will deal with advanced issues in 
inferential statistics. Topics such as Analysis of Variance, 
Factor Analysis, Multivariate Regression, and 
Logit/Probit models will be covered and contrasted with 
other statistical tools and techniques. A case study 
approach will be used illustrating applications of these 
statistical techniques in institutional research. *Open to 
those who have completed the introductory workshop Saturday 
afternoon or who have an equivalent background 
Pre-Conference Workshop 


9:00 - noon 

Chartiers - Mezzanine Level 
Jim Fergerson 

Director of Institutional Planning & 

Analysis 

Bates College 

John Pryor 

Director of Undergraduate Evaluation 
& Research 
Dartmouth College 


Designing and Conducting Web-based Surveys 


This workshop will provide an introduction to designing 
and conducting successful web-based surveys. The 
presenters will address administrative and methodological 
concerns and technological issues. Workshop topics will 
include items such as contacting a sample via email, 
maintaining general security and limiting accesses to the 
survey to pre-selected individuals, guarding against 
multiple responses, and keeping user information 
attached to responses. There will be an introduction to 
setting up an HTML survey form, and an overview of 
some of the software that is available to facilitate a web- 
based survey. The workshop will include 
demonstrations, but is not designed to be hands-on. 
Pre-Conference Workshop 
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NEAIR 27 th Annual Meeting 
Sunday, November 5 th , 2000 



Conference Program 



Noon - 1:30 pm 


Lunch on your own 


1:30 - 4:30 pm 

Traders - Mezzanine Level 
Craig Clagett 

Vice President Planning, Marketing, and 
Assessment 

Carroll Community College 
Michelle Appel 

Director of Institutional Research 
Carroll Community College 


Office Management and Information Dissemination 


Strategies for New Directors of Institutional 


Research 

Designed for institutional researchers who have recendy 
become directors, this workshop focuses on office 
management strategies and techniques for effective 
information dissemination. Topics covered include 
environmental scanning, office staffing, staff incentive 
and recognition programs, office project management 
systems, principles of tabular and graphical data 
presentation, print and electronic reporting. 
Pre-Conference Workshop 


1:30 -4:30 pm 

Brigade - Mezzanine Level 
Stephen R. Porter 
Director of Institutional Research, 
Wesleyan University 

Paul D. Umbach 

Graduate Research Assistant 

University of Maryland, College Park 


Survevs of Students and Facultv: Using Good 
Practices and the Internet to Lower Costs and 
Increase Response Rates 

This workshop explains how to combine good survey 
practices with easy to learn Internet technologies to 
enable institutional researchers to conduct quick and low- 
cost Internet surveys with high response rates. The 
workshop covers topics such as the pros and cons of 
paper and electronic surveys, the skills and software 
needed for electronic surveys, and survey administration 
over the web. 

Pre-Conference Workshop 


5:00 - 6:00 pm 

Ballroom 3 - Mezzanine Level 
David Smallen 
Director of ITS 
Hamilton College 

David is the recent recipient of the 
Educause Leadership Award 


Maintaining Our Bridges - What Do We Reallv 


Know About IT? 

Information technologies are a part of the critical 
connecting infrastructure of our campuses and 
increasingly a center of attention at the highest levels of 
our institutions. We’re wired up, unplugged, webified, 
informated, reengineered, e-everythinged. We’ve shifted 
paradigms, danced with devils, gone the “distance”, and 
managed transitions, quality, and customer relationships, 
And vet, do we reallv know what it takes to sustain our 
technology-rich environments? 

Opening Plenary Session 


Immediately following plenary 
session 

King's Garden North 

King's Garden South and Bateau 

Mezzanine Level 


President's Reception sponsored bv Principia 
Products 

Banquet and Entertainment sponsored bv the Center 
for the Studv of Higher Education at The 
Pennsvlvania State Universitv 
Chamber Music provided by IL Quattro 
Cash Bar 
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NEAIR 27 th Annual Meeting 
Monday, November 6 th , 2000 



Conference Program 



8:00- 11:00 am 

Ballroom 4 - Mezzanine Level 


Conference Registration 


7:15 -8:30 am 

Ballroom 4 - Mezzanine Level 
Ellen Kan are k 
Vice President 

Applied Educational Research, Inc. 


Continental Breakfast sponsored Peterson f s 

Concurrent Special Interest Groups 

Those interested in one of the special interest groups may 
pick up breakfast and take with them to the sessions. 


Hailin Zhang 

Data Specialist, Institutional Research 
University of Massachusetts, Boston 


In addition, there will be several table topics at breakfast: 
ASO Users 

First Year in Institutional Research? 

New to IR? Join one of your fellow colleagues in 
discussing joy, sorrows, successes and failures of your first 
year in a new profession. 


7:30 - 8:30 am 

Brigade - Mezzanine Level 
Margaret K. Cohen 
Assistant Vice President for 
Institutional Research 
George Washington University 


Banner Users Special Interest Group 

This informal session provides an opportunity to meet 
other Banner Users, discuss problems, and share 
solutions. It is an open forum where all who are 
interested have the opportunity to set the agenda. 
Everyone — novice and veteran Bannerites — are welcome. 
SIG 


7:30 - 8:30 am 

Rivers - Mezzanine Level 

Emily Thomas 

Director of Planning and Institutional 
Research 

SUNY Stony Brook 


PeopleSoft Users Special Interest Group 
SIG 


7:30 - 8:30 am 

Traders - Mezzanine Level 
C. Anthony Broh 
COFHE 


COFHE 

COFHE members will meet for a SIG. 


7:30 - 8:30 am 

Kang's Terrace - Mezzanine Level 
Michelle Appel 

Director of Institutional Research 
Carroll Community College 


Datatel Users Group 
SIG 


8:30 - 9:15 am 

Brigade - Mezzanine Level 
John Pryor 

Director of Undergraduate Evaluation 
and Research 
Dartmouth College 


A Diversitv Needs Assessment for Staff 
A three-year long process prefaced the administration of 
this diversity tool at a small private liberal arts college. The 
presentation will outline the creation of this NEAIR 
research grant funded tool — including the many 
discussions, obstacles, re-directions, frustrations and 
triumphs along the way to getting the support for the 
project. Results of the survey will be shared along with 
the reactions to those results. 

Research Paper 
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Conference Program 



8:30 - 9:15 am 

Rivers - Mezzanine Level 

Meihua Zhai 

Director of Institutional Research 
West Chester University of PA 

Jeff Himmelberger 

Coordinator of Institutional Research 

Clark University 

Shuqin Guo 

Coordinator of Evaluation and 
Research 

Walden University 


Using Multiple Projection Models to Fit Different 


Student Populations 

Enrollment projection is becoming one of the major tasks 
in institutional research. Developing a best- fitting 
enrollment projection model has been a major challenge 
for IR researchers. This panel will discuss the pros and 
cons of three different projection models used in three 
different types of institutions. Three different enrollment 
projection models in Excel will also be shared during this 
panel discussion. 

Panel 


8:30 - 9:15 am 

Charders - Mezzanine Level 
Edward J. Torpy 
Sales Engineer 
SPSS Inc. 


SPSS Answer Tree and Clementine for Data Mining 


Data mining (the process of discovering meaningful new 
information in large amounts of data) will be introduced, 
including a discussion of how it differs from traditional 
statistics. A demonstration of SPSS’ leading data mining 
products (Answer Tree and Clementine) will illustrate the 
benefits of data mining to institutional researchers. 
Vendor Showcase 


8:30 - 9:15 am 

Traders - Mezzanine Level 
John L. Yeager 

Associate Profess or /Administrative 
and Policy Studies Department : 

University of Pittsburgh 

Glenn M. Nelson 

R. Tony Eichelberger 


The Development and Utilization of a School 
Benchmarking System for Management 


Improvement 

This is a description of a four-year school benchmarking 
project to improve school management. The 
development of the school level process, data 
requirements and collection issues and utilization issues 
are discussed. The data requirements and utility of this 
process are also examined from a department perspective. 
Research Paper 


8:30 - 9:15 am 
Duquesne - Lobby Level 
Ann H. Dodd 

Senior Consultant, Center for Quality 
and Planning 

Carol Everett 

Associate Director, Center for Quality 
and Planning 

Dan Nugent 

Management Information Associate 
Pennsylvania State University 


Measuring Quality Improvement: A Scorecard 


Approach 

As teamwork becomes an integral part of the way we do 
our work, it is critically important to be able to measure 
the success of team initiatives. The presenters will provide 
information about Penn State’s Quality Scorecard and 
team database, a unique approach to measuring and 
sharing the results of teamwork. 

Workshare 
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Conference Program 



8:30 -9:15 am 

King's Terrace 

Kathleen Keenan 

Director of Institutional Research 

Massachusetts College of Art 


Getting Started in Financial Aid Research 


This workshare will present some strategies employed by 
an institutional research office to improve the quality and 
availability of financial aid data for public information and 
institutional planning at a small public college. The 
discussion will include general and technical issues, 
analytic procedures, and results of some specific projects. 
Workshare 


9:25 - 10:10 am 

Brigade - Mezzanine Level 

Janet Nickels 

Office of Institutional Research 
Carroll Community College 

Barbara Livieratos, Howard 
Community College 

Bob Lynch, Montgomery College 

Koosappa Rajesekhara, Community 
College of Baltimore County 


We Know What They Did Last Summer: A Survey of 


Summer Students at Four Community Colleges 


Students enrolled in summer courses at four Maryland 
community colleges were surveyed about their opinions 
and perceptions of the college, and their course- 
scheduling preferences. Analysis focused on those 
students who normally attend four-year institutions during 
the regular academic year and their comparison of the 
community college with their “home” institution. 
Research Paper 


9:25 - 10:10 am 

Rivers - Mezzanine Level 

David Brodigan 

GDA Research 


The Colleges Students Choose and How They 
Decide 

Data from surveys conducted over the last five year for 
two dozen colleges and universities have been combined 
into a single database that has yielded new insights into the 
thinking of prospective college students as they choose 
among six different categories of colleges and universities. 
What kinds of students choose the most selective liberal 
arts colleges, other liberal arts colleges, large private 
research universities, smaller private universities, public 
flagships, and regional public colleges and universities? 
What kinds of institutions are in competition with each 
other and for which students? 

Workshare 


9:25 - 10:10 am 
Traders - Mezzanine Level 
Mary Louise Gerek 
Institutional Research Analyst 

Phyllis Ladrigan 
Professor of Psychology 
Nazareth College 


In-Class Projects: Using Students to Increase IR 


Resources 

To assist a Classroom Utilization CQI (Continuous 
Quality Improvement) team in determining and planning 
optimal instructional space utilization, the students in an 
Environmental Psychology course inventoried 40 available 
classrooms on campus as a term project. This is a case 
study of cooperation between the IR Office, 
administrative offices, faculty, and students to build a 
creative solution to a shortage of person power. 
Workshare 
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9:25 - 10:10 am 

Charders - Mezzanine Level 
Victor Berutti 
Vice President, Products 
Principia Products, Inc. 


Remark Product Demonstration 

Principia will demonstrate and discuss software tools used 
by IR professionals to quickly and economically capture 
data for their research studies. The Remark Office OMR, 
Remark Web Survey, and Remark Classic OMR software 
will be demonstrated during this session. These products 
are widely used in IR departments to capture data from 
both paper and web- based surveys. 

Vendor Showcase 


9:25 - 10:10 am 
Duquesne - Lobby Level 
Michelle Appel 

Director of Institutional Research 
Carroll Community College 


What’s Happening in the Classroom? Using 
Information about the Teaching and Learning 
Environment in Institutional Effectiveness 
Assessment 

Assessing the teaching and learning environment requires 
not only outcomes assessment but also assessment of the 


Craig Clagett 

Vice President, Planning, Marketing 
and Assessment 


processes by which outcomes are achieved. This paper 
describes a survey which collected data, section by section, 
on instructional methods, course requirements, and 
assessment methodologies. This information was 
integrated into the institutional assessment plan. 

Research Paper 


10:10 - 10:30 am 
Ballroom 4 


Break 


10:30 - 11:10 am 
Brigade - Mezzanine Level 
Ellen Kanarek 
Vice President 

Applied Educational Research, Inc. 


Developing a Web Version of the College Board’s 
Admitted Student Questionnaire 

This workshare will discuss a pilot effort to translate the 
College Board’s ASQ onto the Web. Each of the three 
pilot colleges experienced different problems. The 
discussion will cover the most challenging aspects of 
developing the survey itself, as well as issues that arose 
once the site went live. 

Workshare 


10:30 - 11:10 am 

Traders - Mezzanine Level 

Karen W. Bauer 

Assistant Director of Institutional 
Research and Planning 
University of Delaware, 

NEAIR Past-President 


Select Findings from the UDAES Longitudinal Study 

This presentation describes the research design and select 
findings from the longitudinal study, UDAES, University 
of Delaware Academic Experiences Study. Funded 
through the National Science Foundation, this project 
examines the effectiveness of the Undergraduate Research 
program and its educational effects on students and 
faculty. Finding related to student demographics and 
growth will be shared. 

Research Paper 
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10:30- 11:10 am 

Rivers - Mezzanine Level 

David Wright 

Associate Professor 

Marsha Krotseng 
Vice Provost 

West Liberty State College, 

Former AIR President 


Assessing Outcomes for School of Business Majors 


Using a Primary Trait Analysis 


This paper discusses the development and implementation 
of a student outcomes assessment program for School of 
Business Administration majors at a public baccalaureate 
institution. Specifically, it describes the creation and 
successful use of a Primary Trait Analysis instrument 
during a six- month period. Highlights include a 
description of the process, findings from the pilot, lessons 
learned, and recommendations. 

Research Paper 


10:30 -11:10 am 

King’s Terrace - Mezzanine 

Arthur Kramer 

Director of Institutional Research 
New Jersey City University 


Creation of a Scale to Measure Faculty Development 


Needs and Motivation to Participate in Development 


Programs. 

This paper discusses a faculty survey. Faculty were 
surveyed to: 1) Assess satisfaction with current 
development activities and policies; and 2) Establish a 
foundation for a scale to assess factors that motivate 
faculty to participate in development activities. Results 
revealed general satisfaction and a factor concerned with 
administrative recognition and communication of faculty 
achievement. 

Research Paper 


10:30- 11:10 am 

Chartiers - Mezzanine Level 

Michael J. Strada 

FACDIS Co-Director and Professor 
West Virginia University 


Assessing a Decade of Assessment and Faculty 
Resistance to it 

The Institutional Research literature includes the belief 
that assessment works best when faculty-driven. 
However, exclusive reliance on “hard data” to measure 
student “outcomes” fails (in the eyes of most instructors) 
to satisfy their concerns about relevance, validity, and 
significance. More attention to the ancillary role of “soft 
data,” as well as the assessment of pedagogical “process 
and content” — in addition to standard pedagogical 
“outcomes” — can enhance faculty confidence in 
assessment. And where should this quest for “soft data,” 
plus pedagogical “process and content” begin? With the 
misunderstood course syllabus as a rich source of “soft 
data.” 

Research Paper 
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10:30- 11:10 am 
Duquesne - Lobby Level 
Kathleen Rottier 
Senior Research Analyst 
College of Southern Maryland 

Yun Kim 

Office, Planning and Research 
College of Southern Maryland 

Gayle Fink 

Director Planning and Research 
Anne Arundel Community College 

Oyebanjo Lajubutu 

Director of Institutional Research 

Harford Community College 

Jean Frank 

Senior Research Analyst 
Howard Community College 


Getting Hit with an IT System Change and Surviving 


the Impact on Institutional Research Functions 


Seven crises that had to be overcome by institutional 
researcher in order to survive “The System Change” are 
the focus of this workshare. Concrete strategies to assess 
reliability, complete mandated reports, overcome security 
challenges, and continue institutional research activities 
during an information system change will be discussed. 
Workshare 


11:20 - noon 

Chartiers - Mezzanine Level 

Mitchell S. Nesler 

Director of Research, Academic 

Programs 

Amanda M. Maynard 
Regents College 


Curriculum Review at a Virtual University: An 


External Faculty Panel Approach 

Measuring program effectiveness is an important part of 
ensuring academic excellence in higher education, 
especially for institutions serving students at a distance. 
This paper presents the Regents College model for 
reviewing curriculum structure and program objectives, in 
the context of Biology. Process, challenges, and outcomes 
will be discussed. 

Research Paper 


11:20 - noon 

Brigade - Mezzanine Level 
Linda Strauss 

Director, Penn State Learning Edge 
Academic Program 
Penn State University 

J. Fredericks Volkwein 

The Pennsylvania State University, 

NEAIR President 


Institutional Influence on Student Learning and 


Growth: A Response to Accountability and 


Accreditation Forces in Two and Four Year 


Institutions 

Pascarella’s (1985) General Causal Model serves as a 
conceptual framework to examine the institutional 
characteristics and environments contributing to student 
learning and growth at two and four year institutions. The 
study utilizes a multicampus database with 8,405 students. 
Student learning is measured through self-perceptions and 
faculty perceptions (cumulative grade point average). 
Research Paper 
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11:20 - noon 

Rivers - Mezzanine Level 
Corby A. Coperthwaite 
Director of Planning, Research and 
Assessment 

Marcia Jehnings 

Director, Social Sciences Division 
Manchester Community College 


Implementing a Program of Outcomes Assessment in 


the Land of Steadv Habits 

For years this community college talked about assessment 
and finally, within the last two years, learning outcomes 
for General Education, Student Affairs, and all Academic 
Programs have emerged. Course based and portfolio 
assessments are being piloted. What changed? How did it 
happen? Where will the College go from here? 

Workshare 


11:20 - noon 

Traders - Mezzanine Level 
Gary Choban 
Vice President 
Innervate 


Facilitating the Use of Assessment Data and 
Documenting Program Impact — A Software Solution 


TracDat — a flexible software solution for managing the 
academic assessment process. For an assessment program 
to be effective, all phases of the assessment process must 
be addressed. TracDat is a software solution that provides 
academic departments with an efficient and reliable 
mechanism for managing the assessment process. 

Vendor Showcase 


11:20 - noon 

King's Terrace - Mezzanine Level 
Kenneth R. Ostberg 
Regional Director 

National Student Loan Clearinghouse 


Using Enrollment Search to Enhance Effectiveness 


Institutional researchers can now use Enrollment Search 
to study the migratory patterns of applicants for admission 
and ex-students as they move through the higher 
education system. 

Vendor Showcase 


1 1 :20 - noon 

Duquesne - Lobby Level 
Carol Trosset 

Director of Institutional Research 
Grinnell College 


Usinff Qualitative Analytical Methods for 
Institutional Research 

Statistical analysis is the stock-in-trade for institutional 
research, but the field can also benefit from qualitative 
methods. Trosset, a cultural anthropologist, will share 
several qualitative analyses from her work at Grinnell 
College, explain the techniques involved, and discuss ways 
in which these methods can enhance research efforts. 
Research Paper 


Noon - 2:00 pm 

Ballroom 1 - Mezzanine Level 


Luncheon and Business Meeting 
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2:00 - 3:30 pm 

Brigade - Mezzanine Level 
Stephen Thorpe 
Assistant Provost 
Drexel University 


Online vs. Paper Surveys: A Comparison of 
Methodologies 

The use of online surveys vs. traditional paper methods is 
becoming an increasingly popular approach for campus- 
based research activities. The panelists, each of whom 
have conducted several online studies, will discuss the 


Jim Fergerson 

Director of Institutional Planning and 

Analysis 

Bates College 


advantages and disadvantages of web- based surveys, and 
their campus-based findings of similarities and differences 
in response rates and potential response bias. 

Panel 


Mark Palladino 
Research Specialist 
Drexel University 




John Pryor 

Director of Undergraduate Evaluation 
and Research 
Dartmouth College 




2:00 - 2:40 pm 

Charders - Mezzanine Level 
Tuan Dang Do 

Assistant Director, Institutional 
Research 


Visual IPEDS 

The purpose of this presentation is to describe our progress 
in using object-oriented languages (especially Visual Basic) 
to create programs to automatically complete IPEDS 
reports (enrollment, age, residence, undergrad transfer, 
residence of first time students and credit hours, so far). 


Robert Yanckello 

Director, Institutional Research 

Central Connecticut State University 


This user-friendly interface tool will eliminate many hours 
of work in IR offices. 

Workshare 


2:00 - 2:40 pm 

Rivers - Mezzanine Level 

Anne Marie Delaney 

Director of Institutional Research 

Babson College, 

NEAIR President-Elect 


Institutional Researchers: Challenges, Resources and 
Opportunities 

This paper presents the results of a study that investigated 
challenges institutional researchers encounter in their 
career; resources for coping with these challenges; and the 
impact of these challenges on engagement in policy. 

Results identify concern about the amount of work, limited 
opportunity for advancement, and producing quality work 
within time constraints as the most prevalent challenges. 
However, those who have a mentor, a strong professional 
network and an independent job structure can more 
effectively meet such challenges and actively engage in 
policy development. 

Research Paper 
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2:00 - 2:40 pm 

Traders - Mezzanine Level 
Emily Thomas 

Director of Planning and Inst. 
Research 

Douglas Panico 

Director of Management Analysis & 
Audit, SUNY Stony Brook 


Financial and Performance Profiles of Academic 


Departments 

This workshare will describe how we created academic 
department profiles that include their resources, their 
outputs, and an analysis of their financial contribution to 
the university. We will present our profile, discuss how the 
data are used, and describe how we solved methodological 
and technical problems. 

Workshare 


2:00 - 2:40 pm 
Duquesne - Lobby Level 
Tracy Polinsky 

Coordinator of Institutional Research 
Butler County Community College 


The IR-COI Connection 

"Quality" has been stimulating self-evaluation, creative 
thinking, and change at institutions for years. Because 
quality efforts are data based and assessment dependent, 
they are appropriate projects for institutional researchers. 
By providing data and encouraging systematic evaluation, 
they can help their colleges to successfully implement 
quality efforts at their institutions. 

Research Paper 


2:50 - 3:30 pm 

Duquesne - Lobby Level 

James Robertson 

Assistant Director, Planning and 

Institutional Research 

Community College of Allegheny 

College 

Julia Peters 


End of Month Reporting at CCAC 

In switching from legacy to Datatel, CCAC lost all 
reporting infrastructure, which Institutional Research 
needed to re-create. This paper describes the end of month 
reporting process for creating various enrollment 
comparisons. Anyone who does reporting may be 
interested. Included are queries, SPSS syntaxes, sample 
Excel worksheets and PDF outputs. 


2:50 - 3:30 pm 

Traders - Mezzanine Level 
Sandra Price 

Director of Institutional Research 
Keene State College 

Dawn Geronimo Terkla 
Executive Director, Institutional 
Research 
Tufts University 


What Would You Do? Ethical Scenarios Illustrating 


AIR’s Code of Ethics 

AIR's Code of Ethics is in the process of being revised. 
Members of AIR's Task Force on Ethics will present a 
series of scenarios depicting ethical dilemmas. Following 
the each scenario the audience will be asked to discuss 
several questions regarding the dilemma using the Code as 
reference. 

Workshare 


2:50 - 3:30 pm 

Charders - Mezzanine Level 
Donald A. Gillespie 
Director of Institutional Research 
Fordham University 


Results of an Exploratory Survey of the Staffing and 


Responsibilities of Institutional Research Offices 


This workshare will present the results of an exploratory 
survey of staffing patterns and responsibilities of 
institutional research offices at selected Catholic institutions 
and plans for a survey of a full range of US colleges that 
might examine the amount of time spent on major 
institutional research tasks. 
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2:50 - 3:30 pm 

Brigade - Mezzanine Level 
Cherry Danielson 

University System of New Hampshire 


Change Leadership and the Implications of Culture 


The last twenty years have been riddled with various types 
of change as colleges and universities attempt to position 
themselves for survival and success. While institutions 
have designed strategies for change, the role of leaders in 
the process and their ability to affect outcomes has been 
laden with high expectations. Thus, the relationship 
between leadership and change has emerged as a key 
juncture for scholarly consideration. This literature review 
synthesizes theoretical models, empirical studies, and 
anecdotal writings that address issues of change and 
leadership emanating from both Organization and Higher 
Education literature. 


2:50 - 3:30 pm 

Rivers - Mezzanine Level 

Monica E. Randall 

Associate Director of Policy Analysis 
and Research 

Maryland Higher Education 
Commission 

Geoffrey Newman 
Finance Policy Analyst 
Maryland Higher Education 
Commission 

Elis s a Klein 

Research Director 

Maryland Association of Comm. 

Colleges 


Facilities Planning In the 21 st Century: Developing 


Continuous Education Enrollment Projections For 


Maryland’s Community Colleges 


The purpose of this workshare is to discuss the progress 
that Maryland has made in the development of a 
methodology for projecting noncredit continuing education 
enrollments at Maryland’s community colleges. The 
workshare presenters will discuss the history of the 
development of continuing education enrollment 
projections; the methodology for projecting eligible 
noncredit enrollments; and the policy issues related to the 
development of this model. This workshare will appeal to 
those interested in projecting noncredit continuing 
education enrollments and at those interested in facilities 
planning. 

Workshare 


3:45 - 4:45 pm 

Ballroom 4 - Mezzanine Level 
Tom Mortenson 

Post Secondary Opportunity 


Higher educational opportunity in the human capital 


economy 

■ The human capital economy (income by educational 
attainment) 

■ Social and private investment in human capital 

■ The distribution/redistribution of higher education by 
family income over the last three decades in the U.S 




>■ 
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5:00 - 6:00 pm 


Happy Hour (meet friends and make dinner plans) 


Ballroom 3 — Mezzanine Level ' 


Concurrent Table Topics and Special Interest Groups 


Kit Mahoney, CIRP Survey 


Using the CIRP Surveys for Student Assessment 


Coordinator-UCLA's Higher 


Colleges can collect valuable baseline data on their entering 


Education Research Institute 


students using the Cooperative Institutional Research 
Program (CIRP) Freshman/Entering Student Survey. By 
following-up these same students later with the College 
Student Survey, colleges accumulate comprehensive data 
on their students. A growing number of colleges are using 
these data for accreditation self-studies; satisfying state- 
mandated performance measures and monitoring the 
impact of college on students. The discussion will cover 
practical considerations of using the combination of 
CIRP/CSS for longitudinal assessment. 


Mark Zidzik 


The Babv and the Bath Water: What Data Are 


Director, Research Development 


Important When Profiling Graduate and Professional 


Peterson’s 


Programs 

Given the different perspectives of data providers and 


Rocco Russo 


collectors and information suppliers and users, the question 


Vice President of Research 


of what data are most important when researching 


Peterson’s 


postbaccalaureate study opportunities has many answers. 
This table topic, facilitated by Research staff from 
Peterson’s, will feature discussion of the relative merits of 
data that are collected in each of the following areas: 
enrollment, faculty, research, degrees, academic subject 
areas, requirements, completions, and financial aid. 


Valerie S. Rogers 


Selecting Peer Institutions 


Assistant to Director, Office of 


With the recent changes in the Carnegie Classifications this 


Institutional Research 


table topic will discuss a University’s process in re-defining 


University of Connecticut 


its peer base institutions. What is an appropriate number 
of peers? What factors should be considered when 
defining a peer group? Others are encouraged to share 
their experiences in peer selection 


Pam Roelfs 


Performance Indicators: The Good, The Bad, and 


Director of Institutional Research 


The Uglv 


University of Connecticut 


General discussion of indicators of effectiveness, efficiency, 
and “success” for colleges and universities will be the main 
purpose of this table topic. Which performance indicators 
are good? Which ones are bad? How ugly have been the 
definition, application, measurement, and interpretation of 
them? Discussion will focus on indicators used in 
institutional comparisons. 
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Christopher Hourigan 

Assistant Director, Planning, Research 

and Evaluation 

William Paterson University 


Collaboration between Institutional Research and 
Academic Departments 

In addition to providing information and analysis to the 
administration regularly, institutional researchers can also 
help an institution to work towards its mission by serving 
as a resource for academic departments. This table topic 
will be a discussion about how institutional research offices 
can make valuable contributions to academic departments 
and will feature examples of the work that the Office of 
Planning, Research and Evaluation has done for the 
academic departments at William Paterson University. 

Special Interest Groups 


Jason Casey 

Brigade - Mezzanine Level 


HEDS 


Linda Junker 

Traders — Mezzanine Level 


Catholic Colleges 


Peter Parnell 

Rivers - Mezzanine Level 


SUNYAIRPO 
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7:15 - 8:30 a.m. 

Ballroom 4 - Mezzanine Level 


Continental Breakfast sponsored bv George Dehne & 
Associates 


8:00 - 8:40 am 

Traders - Mezzanine Level 
Robert K. Toutkoushian 
Executive Director, Office of Policy 
Analysis 

University System of New Hampshire 


A Comparison of Faculty in Regular Versus Non- 
Regular Academic Positions 

This study uses data from the NSOPF:93 national survey 
of faculty to examine the satisfaction and relative 
compensation of faculty employed in regular versus non- 
regular academic positions. For the purpose of this study, 
faculty are broken into four categories (tenure/ tenure- track 
vs. non- tenured, full-time vs. part-time). Descriptive 
statistics and multivariate regression techniques are then 
used to compare faculty in these groups on the basis of 
their background characteristics, satisfaction with academic 
employment, and compensation. 

Research Paper 


8:00 - 8:40 am 

Rivers - Mezzanine Level 
Dawn Geronimo Terkla 
Executive Director, Institutional 
Research 
Tufts University 


New Technology and Student Interaction with the 
Institution 

This paper examines how prospective students as well as 
current undergraduates are using electronic communication 
to interact with various campus constituencies. Findings 
show that students extensively use e-mail and IRC to 
communicate with friends and colleagues, but the use of 


Gordon J. Hewitt 

Assistant Director of Institutional 

Research 

Tufts University 


these mediums — as well as other interactive Web-based 
mediums — to communicate with faculty and staff and 
obtain admissions information is much less. 

Research Paper 


8:00 - 8:40 am 

Brigade - Mezzanine Level 
Kelli Armstrong 

Director of Institutional Research 
UMass President’s Office 


Keeping It Private or Bringing It Public: Careers in IR 

Have you ever wondered what it was like to work “on the 
other side?” Sessions at institutional research conferences 
are often divided among public and private institution lines. 
Hot issues that are pressing for colleagues on public 
campuses may not be so for institutional researchers at 


Becky Brodigan 

Director of Institutional Research 
Middlebury College 


private colleges (and vice versa.) This session is designed 
to be an open discussion about career paths in institutional 
research. The panelists will speak from personal 
experiences about crossing the border between private and 
public institutions, and moving into areas beyond 
traditional institutional research work. 

Workshare 


8:00 - 8:40 am 

Traders - Mezzanine Level 
Meihua Zhai 

Director of Institutional Research, 
Office of Planning & Analysis 


The Impact of Remedial English Courses on Student 
College-Level English Performance and Persistence 

The impact of remedial English class on student 
persistence and performance in their college-level English 
was studied. Retention rates and percentages of students 
who passed their college-level English were compared 


Jennie Skirl 

Associate Dean of Arts & Sciences 
West Chester University of PA 


between remedial and non-remedial course takers whose 
SATV were below 500 (550 after recentering). Student 
course grades from fall 1992 to spring 200 were used in this 
.study. 

Research Paper 
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8:00 - 8:40 am 

Duquesne - Lobby Level 
David X. Cheng 

Assistant Dean for Research and 

Planning 

University 


Student Self-Perceived Gain Scales as the Outcome 


Measures of Collegiate Experience 

This study attempts to articulate student collegiate 
experience using self reports and to construct the gain 
scales that can be used as the outcome measures in an 
institution’s overall assessment efforts. 

Research Paper 


8:50 - 9:30 am 

Brigade - Mezzanine Level 

Ronald Zaccari 

President 

West Liberty State College 


A Presidential Conversation: Collaborating for 


Change 

Working together, institutional researchers and presidents 
can provide a solid force for change and enliven the 
strategic planning and management of their colleges and 
universities. This dialogue between institutional researchers 
and a college president will explore ways to foster such 
opportunities and consider a variety of issues, including 
how Institutional researchers can creatively assist presidents 
and ways in which presidents can effectively employ their 
institutional research offices, a public baccalaureate 
institution over a four-year period. Numerous changes 
resulting from the plan are highlighted. 


8:50 - 9:30 am 

King's Terrace - Mezzanine Level 
Michael J. Dooris 
Director, Planning, Research & 
Assessment, Center for Quality & 
Planning 

Louise E. Sandmeyer 

Executive Director, Center for Quality 

and Planning 

Pennsylvania State University 


Faculty & Staff Surveys: Insight for Improvement 


At Penn State, university, college, and department 
improvement efforts can draw from a centrally assembled 
package of tools — such as surveys and exit interviews — to 
gain insight into faculty and staff opinion. The presenters 
will share examples from Penn State, and invite participants 
to discuss approaches at their institutions. 

Workshare 


8:50 - 9:30 am 

Rivers - Mezzanine Level 

Mitchell S. Nesler 

Director of Research, Academic 

Program 

Regents College 

Roy G. Gunnarsson 


What Facilitates or Inhibits Adults from Participating 


in Adult Education? An Analysis of the National 


Household Education Survey. 

This study was designed to examine the self-reported 
barriers adults face to accessing adult education, their 
motivations for participating in adult education, and the 
demographic characteristics associated with these factors. 
NHES:95 data were analyzed to address these questions. 
Research Paper 
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8:50 - 9:30 am 

Traders - Mezzanine Level 
Tsuey-Ping Lee 

Assistant for Institutional Research 
University at Albany, SUNY 

Chisato Tada 

International Student Advisor 
University at Albany, SUNY 


To Show How We Care: Combining Web-Based 


Technology and International Student Needs 
Assessment 

The purposes of this research are to assess international 
student needs and to experiment with web-based survey 
techniques. This research paper not only analyzes the 
results based on the degree level of international students, 
cultural background, academic major and length of stay in 
US, but also details the basic survey research issues and 
complexities of conducting a web-based, and traditional 
paper surveys. This study will present the detailed survey 
processes, the data, the research results and the application 
of the results. 

Research Paper 


8:50 - 9:30 am 

Charders - Mezzanine Level 
Kevin B. Murphy 
Institutional Research Analyst 
University of Massachusetts, Boston 


Developing an Analvsis of Outcomes for the Writing 


Proficiencv Requirement 

This is a case study of the process of developing an analysis 
of outcomes for the writing proficiency requirement. It 
will focus on the role of the institutional researcher in 
question formulation, identifying what is currently feasible, 
and preparing to better answer the question in the future. 
Research Paper 


8:50 - 9:30 am 

Duquesne - Lobby Level 
Karl Boughan 

Coordinator of Institutional Research 
Prince George’s Community College 


Through the Development Maze: Remedial Program 


Complexitv and Student Progress at a Large, 

Suburban Communitv College 

Unlike most past developmental program research 
emphasizing the external correlates of remedial success, this 
community college case study focuses instead on program 
configuration and its interaction with the credit 
instructional process and new student expectations of 
college. Cluster analysis is used to clarify the tangled web 
of forces at work, sorting a cohort of recent fall-entering 
remedial students into discrete “developmental strategy” 
groups, each representing a unique set of student 
behavioral responses to the remedial process and a unique 
remediation outcome pattern. 

Research Paper 


9:40 - 10:20 am 

Brigade - Mezzanine Level 

Marsha V. Krotseng 

Vice Provost 

Ronald Zaccari 
President 

West Liberty State University 


The Transformational Power of Strategic Planning 


Strategic planning is vital to the effective management of 
colleges and universities. It also is integral to institutional 
change. This case study demonstrates the critical 
connection between strategic planning and institutional 
transformation by tracing the strategic planning process for 
a public baccalaureate institution over a four-year period. 
Numerous changes resulting from the plan are highlighted. 
Research Paper 
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9:40 - 10:20 am 

Rivers - Mezzanine Level 

Richard J. Reeves 

Senior Research and Planning 

Associate 

Cornell University 


Data Mining Basics: What is it and whv use it? 


Intended for institutional researchers interested in 
developing their own data-mining system, this presentation 
will briefly cover the following topics: what research 
methods constitute data-mining, how it can be used to 
improve enrollment management, a brief comparison of 
data-mining to traditional statistics, and the evolution of 
data-mining. The presenter will then discuss the 
components (technology and personnel) necessary to create 
a functional data-mining system. 

Workshare 


9:40 - 10:20 am 

Charders - Mezzanine Level 
Stephen R. Porter 
Director of Institutional Research, 
Wesleyan University 

Paul D. Umbach 

Graduate Research Assistant 

University of Maryland, College Park 


We Can’t Get There in Time: Assessing the Time 


between Classes and Classroom Disruptions 


This workshare describes and analyzes the time between 
classes problem at the University of Maryland. Using 
facilities and course scheduling data in combination with 
student survey data, we discovered that many students had 
distances to travel between classes that take longer than the 
allotted ten minutes. The survey indicated that students 
reacted by leaving class early and skipping class altogether. 
Reasons for having such a class schedule ranged from 
problems registering for a particular course to a desire for a 
compact schedule. 

Workshare 


9:40 - 10:20 am 

Charders - Mezzanine Level 

Robert Morse 

US News and World Report 

Peggye Cohen 

George Washington University 
Moderator 


The U.S. News College Rankings 

A detailed explanation and discussion of the methodology 
changes made in the "America's Best Colleges" rankings 
published on September 1, 2000. U.S. News views on the 
September 2000 Washington Monthly article "Playing With 
Numbers." An opportunity to ask questions about the 
rankings. 


10:30 - noon 

Ballroom 3 - Mezzanine Level 
Dawn Geronimo Terkla, Incoming 
AIR President and Executive Director 
of Institutional Research, Tufts 
University 

Jennifer Brown, Director of 
Institutional Research and Policy 
Studies, University of Massachusetts, 
Boston 

Mark Putnam, Director of University 
Planning and Research, Northeastern 
University and Chair, NPEC 
Committee on College Costs 


What’s Happening in Washington: An update on 


Institutional Research Issues from a National 


Perspective 

Members of various NPEC and AIR committees will 
report on the latest happenings regard Student Outcomes, 
College Costs and a variety of other issues. 

Plenary Session 
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